Databricks Shows How to Build Durable Agents with Temporal and Lakebase, but the Demo Exposes the Hard Part

Databricks published a reference implementation on September 8 showing how to Build durable agents with Temporal and Lakebase despite worker crashes, retries, and multiday reviews. The system applies this architecture to personal-loan underwriting, where losing one completed check can corrupt a decision trail.

The important change is not another agent framework or a larger model. Databricks and Temporal have divided agent state between two systems with different responsibilities. Temporal preserves control flow, while Lakebase exposes operational data that applications, reviewers, and analysts can query.

That division also creates the central tension. Durable execution can recover recorded work, but it cannot make every external effect exactly once. The application still needs stable identifiers, guarded database writes, policy-version rules, and reconciliation when components disagree.

Databricks Turns Agent Durability Into a Testable System

The reference implementation treats an agent as a long-running business process, not a temporary chat session.

The reference implementation follows a loan request from evidence collection through a human decision. Its agent performs credit, income, debt-to-income, and underwriting-policy checks as separate operations.

The example uses mocked applicants and providers, so it does not process real loan applications. That choice keeps the experiment focused on execution behavior instead of lending-model performance.

One sample applicant has a 665 credit score and one non-material delinquency flag. The agent collects the evidence, evaluates purpose-specific policy thresholds, and generates a recommendation. It cannot make the final lending decision.

An underwriter must approve, deny, or request more information. That last option extends the same case into another agent turn, with the earlier evidence and reviewer rationale preserved.

The scenario is deliberately harder than a single model request. A worker can stop after several checks finish. A database write can commit before its completion reaches Temporal. A reviewer can leave the case open for days.

A stale browser can also submit an outdated command after the case has moved forward. Meanwhile, underwriting policy can change without a corresponding application deployment.

These situations produce six practical requirements: recovery, controlled retries, durable waits, operational visibility, runtime governance, and audit history. A transcript alone cannot satisfy them.

A transcript records messages, but not necessarily the complete control flow. It does not automatically show which operation completed, which result was accepted, or which command advanced the process.

The implementation therefore gives each loan run a Temporal Workflow. A Workflow is durable control flow whose recorded history allows another worker to reconstruct its state.

Calls to models, databases, and underwriting tools run as Activities. An Activity is a retryable operation whose result can be recorded in the Workflow’s Event History.

Reviewer responses arrive as Signals, which are asynchronous commands delivered to an open Workflow. Temporal can retain that wait without reserving a worker process for several days.

React and FastAPI handle the user-facing application. They start runs, display evidence, list cases, and submit review decisions. Temporal Cloud stores execution history and dispatches tasks to workers.

Lakebase Postgres holds the application-facing projection. A projection is a queryable representation of current workflow state, built from updates produced during execution.

Unity Catalog remains the source for underwriting rules. A continuous synced table makes those rules available through Lakebase, so workers can read updated policy without a code deployment.

This architecture makes agent failure behavior visible and reproducible. It turns durability from a general promise into a set of specific recovery and consistency contracts.

However, the demo does not collapse those contracts into one database. Temporal and Lakebase remain separate systems, and the gap between them drives the harder engineering questions.

Agent Memory Is Not Execution State

A durable agent needs recorded decisions about control flow, not merely stored messages or retrieved memories.

Many agent systems describe persistence as memory. They save a conversation, retrieve earlier documents, or place an intermediate result in a database. Those capabilities help models recover context, but they do not reconstruct execution.

Suppose a worker finishes a credit check and then exits. A replacement must determine whether the result was recorded, whether another attempt is safe, and which operation should run next.

That is a control-flow problem. It includes scheduled activities, completed results, timers, accepted human commands, retry attempts, and the current review round.

Temporal stores this information in an ordered Event History. During replay, workflow code consumes those recorded events and rebuilds variables such as evidence, token usage, and review state.

A recorded Activity result returns during replay instead of executing again. Therefore, a completed credit check or recorded model response remains fixed for that workflow execution.

An unrecorded completion presents a different case. A model provider might finish processing a request just before the worker loses connectivity. If Temporal never receives the result, it can schedule another attempt.

That limitation matters because model calls are neither free of side effects nor guaranteed to be deterministic. A second response can differ from the first, even with identical input.

The demonstration assigns retry policies to individual operation types. Model Activities permit up to four attempts within a three-minute schedule-to-close window.

Tool Activities permit up to three attempts with a 60-second start-to-close timeout. Lakebase Activities permit up to five attempts with a 15-second start-to-close timeout.

These figures describe the example configuration, not universal production defaults. Teams must set retry limits according to provider behavior, latency targets, failure modes, and downstream consequences.

Temporal’s durable execution model addresses process recovery by preserving the history needed for replay. It does not automatically make a payment, email, database mutation, or model request safe to repeat.

Each external operation needs an idempotency contract. Idempotency means repeated attempts converge on one intended outcome instead of producing duplicate effects.

The loan demo constructs this contract with deterministic identifiers. A run, message, tool call, event, review round, and reviewer decision each receive a stable identity.

Postgres primary keys and unique constraints prevent retries from creating unlimited copies. Upserts allow a repeated attempt to target the same logical row.

Guarded updates add another layer. They permit only valid state transitions, such as moving a pending tool call to completion without reopening a terminal record.

Yet even a guarded update needs careful interpretation. PostgreSQL can affect zero rows without raising an error when the target already reached a terminal state.

The Databricks article acknowledges that the current Activity wrapper does not always turn that zero-row outcome into a failure. Production code should inspect stored state before treating it as harmless.

This detail separates a useful engineering reference from a finished production pattern. Durable orchestration provides the recovery mechanism, while application developers still define safe business semantics.

The same principle applies outside underwriting. Payment providers need idempotency keys, email systems need stable message identifiers, and unsupported tools need reconciliation records.

Teams building an internal agent also need a searchable evidence layer. A structured engineering knowledge base can help people inspect documentation, decisions, and technical context around those workflows.

The larger lesson is precise: memory helps a model remember, while durable execution helps a system continue. Production agents usually need both, but the two are not interchangeable.

Build Durable Agents With Temporal and Lakebase by Separating Responsibilities

The design works because Temporal and Lakebase hold different kinds of truth for different consumers.

Temporal owns execution truth. Its history determines which tasks completed, which timers fired, which Signals arrived, and what a replaying worker should do next.

Lakebase owns the current application view. It stores run status, messages, tool evidence, review records, operational events, recommendation metadata, and retry metrics in relational tables.

That division lets the user interface query current cases with ordinary SQL patterns. Reviewers can find pending cases, inspect one recommendation, or compare operational measurements across executions.

Evidence appears before a Workflow closes. Once the policy lookup finishes, the structured result becomes available alongside thresholds, actual values, rule outcomes, rationale, and policy source.

This matters in human-reviewed systems. A reviewer should not wait for the entire process to finish before seeing the evidence behind a recommendation.

The application uses two Lakebase schemas. The agent_ops schema holds operational records, while agent_policy contains a read-only copy of governed underwriting policy.

Every Activity writes rows using identifiers aligned with the Workflow. A retry can therefore update the same logical record while the database projection catches up.

Lakebase does not become part of Temporal replay. That boundary prevents ordinary application queries from deciding workflow execution, but it also means updates are not atomic across both systems.

A Temporal event can be recorded while a Lakebase projection remains temporarily behind. A database write can also commit before Temporal records the corresponding Activity completion.

The architecture accepts this gap and uses eventual consistency. Eventual consistency means separate views can disagree briefly but converge through retries and deterministic writes.

That is a reasonable design for dashboards and case lists. It demands more caution when a database view is used to validate a command affecting business state.

Lakebase contributes familiar Postgres access and application-oriented indexing. Its operational database model also supports agent memory, current state, and feature-serving workloads.

Its compute can autoscale within configured limits. Scale-to-zero can suspend idle compute, although the first query after inactivity can experience activation latency.

Those database features help with uneven agent traffic. They do not eliminate capacity planning, connection limits, pool configuration, or recovery testing.

The policy path is equally important. Unity Catalog stores purpose-specific thresholds, including credit and debt-to-income rules. A continuous synced table presents those values inside Lakebase.

Policy owners can update the source without redeploying the worker or API. A later lookup can read the propagated rules from Postgres.

This avoids embedding every business threshold in application code. It also introduces a policy-timing question that the orchestration layer must answer explicitly.

Should an open case keep the rules applied at its start, or adopt a newer policy during a later turn? Both choices have consequences for consistency, auditability, and customer treatment.

The demo records the applied thresholds and source with the recommendation. That evidence lets reviewers reconstruct which policy informed a particular result.

When Lakebase is unavailable, the sample can use fixture policy and record that fallback path. The article correctly notes that a regulated workflow might instead fail closed.

That choice cannot be delegated to a retry library. Product owners, compliance teams, and engineers must define whether stale or fallback policy is legally and operationally acceptable.

The proposed return path uses Lakebase Change Data Feed. Once enabled, it can capture database mutations and publish them to Unity Catalog-managed Delta history tables.

Databricks says the feed batches changes approximately every 15 seconds. That interval suits retrospective audit and analysis, while the application reads current state directly from Lakebase.

However, the repository only prepares its schemas for that path. It does not contain an observed end-to-end feed run in the target environment.

Building durable agents with Temporal and Lakebase therefore requires three clear boundaries: execution truth, operational truth, and governed analytical history.

The pattern becomes valuable when those boundaries are explicit. It becomes dangerous when teams assume the word “durable” means every component always agrees.

Human Review Exposes the Consistency Tradeoff

The underwriter wait shows why durability must include command identity, stale-state rejection, and independent workflow validation.

After the model produces a recommendation, the Workflow creates a review identifier from the run and current turn. It records the pending review in Lakebase and enters AWAITING_REVIEW.

Temporal then waits for a condition without keeping a worker occupied. The open Workflow can survive process replacement while the human takes time to respond.

The API accepts approval, denial, or a request for more information. It first checks whether Lakebase still shows the relevant review as pending.

It also compares the submitted review identifier with the current review round. A mismatch produces a conflict instead of forwarding an obviously stale decision.

This database precheck improves the user experience, but it is not authoritative. The Lakebase projection can lag behind Temporal, particularly while an Activity retries.

The API therefore sends the command as a Signal, and the Workflow validates it again against execution state. Duplicate or stale decisions are ignored inside the durable control flow.

That second check is essential. A browser tab might remain open while another reviewer advances the case, or an earlier request might arrive after a new review round begins.

An HTTP 202 response only confirms that Temporal received the Signal. It does not mean the Workflow accepted the business decision.

The client must refresh the Lakebase view to observe the resulting state. That distinction prevents transport acknowledgment from being mistaken for loan approval.

When the Workflow accepts a decision, an idempotent Lakebase Activity persists it. Approval or denial completes the run.

A request for more information resumes execution. The reviewer’s rationale becomes a new user message, the turn advances, and the next recommendation gets a new review identifier.

This mechanism gives every review round a stable boundary. It also makes the main tradeoff visible: responsive application state is separate from authoritative execution state.

Teams must design for temporary disagreement. Interfaces should communicate pending commands, conflicts, delayed projections, and rejected stale actions clearly.

Operational dashboards also need to distinguish intentional waiting from failure. A case awaiting an underwriter is healthy, while an Activity stuck in retries needs intervention.

The demo exposes metrics at Workflow, turn, and Activity-attempt levels. Operators can inspect Temporal history, query Lakebase state, and check the worker environment separately.

This separation helps diagnose whether a delay comes from human review, model availability, database authentication, or a failed tool.

It also adds operational surface area. Teams must monitor Temporal, Lakebase, worker deployments, connection pools, synchronization jobs, and the contracts joining them.

Authentication introduces another long-running concern. The Lakebase client uses machine-to-machine OAuth and refreshes its connection pool before temporary database credentials expire.

Without credential rotation, a worker can fail on a predictable schedule even if its Workflow remains recoverable. Durable control flow does not make expired connections usable.

Developers comparing durable agent frameworks should therefore look beyond checkpoint support. They should ask how commands are identified, how side effects deduplicate, and where authoritative state lives.

They should also test stale interactions deliberately. Open two review sessions, advance one, and then submit the older decision.

A correct implementation should reject or safely ignore that command. It should retain enough evidence to explain the outcome afterward.

The loan example is useful because it connects these details to a consequential decision. Human approval is not a decorative pause between model calls.

It is a state transition with identity, authorization, policy context, and audit requirements. That makes it a sharper durability test than a conversational assistant restarting after an error.

The Demo Does Not Prove Production Readiness

Databricks presents a credible architecture, but its own evidence leaves lending compliance, scale, and full data-loop validation unresolved.

The repository reports 21 passing tests covering workflow sequencing, review behavior, OAuth construction, idempotent persistence, metrics contracts, API startup, and worker settings.

A crash-recovery exercise uses a deterministic provider. It stops worker execution and verifies that recorded progress survives after the process returns.

These tests support the narrow durability claim. They show that the example’s control flow and persistence contracts behave as intended under selected failures.

They do not validate the agent as a lending system. Applicant records and provider responses are fixtures, while the default scripted provider avoids a live model dependency.

The repository does not establish lending-model quality, regulatory compliance, production security, regional availability, or performance at scale. Databricks states these limits directly.

The local crash exercise also ran with Lakebase disabled. It isolates Temporal recovery but does not verify recovery across the complete integrated data path.

Likewise, the Change Data Feed path remains an enablement and deployment task. The example prepares the tables, but it does not show observed history arriving end to end.

Change Data Feed was in Public Preview when the article appeared. Preview status matters for teams that require mature support commitments or validated regional coverage.

The database and Workflow do not share a transaction. Stable identifiers and retries provide convergence, but teams still need reconciliation for prolonged or unexpected divergence.

A production reconciliation process should locate Workflows without matching projections. It should also detect committed database effects whose Activity results never entered Event History.

Policy synchronization deserves the same scrutiny. A later run can consume updated policy without deployment, but an open run needs a documented policy-version rule.

Fallback behavior creates another risk. The demo can substitute fixture policy when Lakebase is disabled or a policy row is unavailable.

That behavior helps local development, but silent fallback would be unacceptable in many regulated environments. The system records the fallback, yet policy owners must decide whether execution should continue.

Performance remains untested in the published evidence. Agent workloads can create bursty writes, long histories, large transcripts, and uneven review queues.

Temporal history retention, retry volume, model latency, worker concurrency, database connection limits, and projection updates all affect system behavior.

Lakebase autoscaling can respond to demand, but new connections and warmed data still matter. Scale-to-zero can also trade idle efficiency for wake-up latency.

Security requirements extend beyond TLS and temporary credentials. A real underwriting platform would need strict authorization, sensitive-data controls, retention rules, and audit access boundaries.

The model’s recommendation also needs independent governance. Recording evidence and policy improves traceability, but it does not establish that the recommendation is fair or accurate.

A complete evaluation would test adverse-action reasoning, missing evidence, conflicting provider data, policy changes, and attempts to manipulate tool inputs.

It would also test operational incidents that cross component boundaries. Examples include unavailable policy sync, delayed projections, partial credential rotation, and unavailable external providers.

The architecture should therefore be read as production-shaped, not production-certified. That distinction strengthens the reference because it makes the remaining work visible.

Databricks has shown how to assign responsibility among orchestration, operational storage, and governed data. It has not removed the need to validate every contract under real load.

Three Signals Will Show Whether This Pattern Holds Up

The next evidence should test integrated recovery, policy consistency, and adoption beyond the controlled underwriting demonstration.

The first signal is an observed end-to-end Change Data Feed deployment. The published system prepares operational tables but leaves feed activation and destination verification unfinished.

A successful validation should trace one run from tool evidence through human review into Unity Catalog-managed history. It should also document delay, duplicates, schema changes, and recovery after interruption.

That result would strengthen the claim that operational activity can return to governed analytical history. Continued reliance on an unverified path would weaken the complete data-loop story.

The second signal is a public load and failure study using Lakebase throughout the recovery test. It should include worker crashes, database retries, credential rotation, and projection lag.

Useful results would report completion behavior, retry distribution, stale-command rejection, and the time required for application state to converge.

This would test the architecture as a combined system rather than validating Temporal recovery in isolation. It would also reveal where scaling limits or operational bottlenecks appear.

The third signal is adoption in a real human-reviewed workflow. Lending is only a demonstration, but similar requirements appear in claims processing, procurement, security response, and regulated approvals.

A credible deployment should explain how it versions policy, reconciles state, controls fallback behavior, and audits human intervention. Those practices matter more than the particular model provider.

Evidence from such a deployment would support the article’s central judgment. Durable agents are distributed applications whose failures must be modeled explicitly.

A deployment that reduces durability to conversation checkpoints would point in the opposite direction. It would leave side effects, long waits, and changing policy outside the recovery contract.

For developers, the immediate question is not whether every agent needs Temporal and Lakebase. Many short, read-only tasks do not justify two managed systems and a projection layer.

The question is whether an agent can outlive its worker, change external state, wait for people, or apply policy that changes independently. Those traits create the need for durable execution.

If they describe your system, inspect the agent architecture, reproduce its crash tests, and challenge every retryable side effect. Then test what happens when execution truth and operational truth temporarily disagree.

That is the practical standard for teams trying to Build durable agents with Temporal and Lakebase. Start with one consequential workflow, define each system’s authority, and make recovery evidence part of the product review.