Databricks Backstage Turns FinOps Into One Query, but the Hard Part Moves Upstream

Databricks has completed a three-part Backstage experiment with one striking result: a single query can connect infrastructure ownership to daily Lakebase consumption. The databricks backstage design removes an integration boundary that usually separates platform engineering from FinOps. However, it also exposes a harder dependency on accurate service metadata.

The demonstration joins Backstage’s live software catalog with Databricks billing records without first moving both datasets into a separate reporting system. A FinOps analyst can identify a Lakebase resource, find its owner, and inspect its usage through one SQL statement.

That is more consequential than another dashboard integration. Backstage normally holds operational ownership data, while a warehouse holds cost records. Teams connect those systems through ETL jobs, exports, tickets, and manually maintained mappings.

The new approach replaces much of that movement with federation, which queries data across systems through a shared SQL layer. Yet it does not eliminate the organizational work behind chargeback. Instead, it relocates that work into catalog annotations, resource identifiers, access controls, and billing semantics.

The Databricks Backstage Experiment Reaches Its FinOps Payoff

The third experiment turns Backstage’s ownership graph into a direct entry point for cloud cost analysis.

Databricks and Thoughtworks used Backstage, Spotify’s open-source internal developer portal, as the operational application throughout the series. Backstage maintains a software catalog containing services, components, owners, dependencies, and infrastructure references.

The teams moved the portal’s PostgreSQL state to Lakebase, Databricks’ managed Postgres service. Lakebase separates storage from compute and integrates transactional data with the wider Databricks environment.

Part one focused on database branching. According to the authors, Lakebase produced a database branch in about one second. That made it practical to test Backstage migrations against an isolated copy before changing the production database.

Part two brought the operational database under Unity Catalog, Databricks’ governance layer for data and AI assets. That step made the Backstage database visible through the same control plane used for analytical data.

The third installment applies those earlier changes to FinOps. Its central query combines records from Backstage with Databricks’ system billing table.

The Backstage side uses final_entities, a table containing the catalog’s processed entity records. Each relevant entity includes a databricks/project-id annotation identifying its Lakebase project.

The billing side uses system.billing.usage, which records billable Databricks consumption. The query matches the catalog annotation with usage_metadata.project_id, then filters for Lakebase usage.

The published example groups consumption by Backstage resource name, Lakebase project ID, and date. Its sample output attributes 39.8667 DBUs on April 8, 2026, and 43.6231 DBUs on April 7.

A DBU, or Databricks Unit, is a normalized measure used to represent platform processing consumption. The query reports DBUs rather than a final currency charge.

That distinction matters. The demonstration proves that usage can be attributed to a cataloged resource. It does not establish a complete accounting model for every discount, commitment, adjustment, or shared expense.

Still, it shortens a familiar FinOps investigation. An analyst no longer needs to request a service ownership export before comparing that export with warehouse billing data.

The result also links cost information to the place where developers already discover services. That creates a path toward cost views organized by systems, teams, or products instead of cloud resource names alone.

The series therefore ends somewhere different from where it began. Database branching improved an engineering workflow. Unified governance then made operational state queryable. The final step converts those technical changes into an organizational reporting mechanism.

Why One Query Changes the Platform Team’s Role

The important change is not shorter SQL; it is the removal of a recurring negotiation between platform, data, and finance teams.

A typical internal developer portal answers questions such as who owns a service, which repository contains it, and what infrastructure supports it. A billing system answers how much metered consumption a resource generated.

Those answers often use incompatible identifiers. A developer recognizes a service called checkout-api, while a cloud invoice exposes an account, project, cluster, or opaque resource ID.

FinOps teams address that mismatch with tags, allocation rules, reporting models, and manual review. The FinOps Framework treats allocation as a core capability because unallocated costs weaken accountability and forecasting.

Backstage offers a potential ownership layer for that process. Its catalog already associates software entities with teams and infrastructure. The challenge is connecting those associations to billing records reliably.

The Databricks experiment puts the connection inside the data platform. Lakehouse Federation, a capability for querying external or operational sources without conventional ingestion, exposes the live Backstage catalog beside system billing data.

This approach reduces the need for a dedicated pipeline that copies ownership records into a warehouse. It also avoids waiting for that pipeline’s next scheduled run before resolving a cost question.

Separated compute is central to the proposal. Lakebase can serve Backstage’s transactional requests while another compute resource performs analytical queries against related data.

Databricks describes Lakebase Postgres as supporting autoscaling, scale-to-zero operation, branches, read replicas, and instant restore. It also integrates with Unity Catalog and Databricks Apps.

The architecture aims to protect the portal from the analyst’s workload. A large aggregation should not compete directly with Backstage’s interactive requests merely because both refer to the same underlying state.

That separation changes what platform teams can offer. They can treat ownership and consumption as two views of the same resource graph, rather than two datasets connected after several handoffs.

FinOps analysts gain a direct route from cost to accountable owner. Platform engineers gain evidence about which internal services consume Lakebase capacity. Engineering leaders gain a potential foundation for team-level reporting.

Developers also face more visible accountability. A catalog entry stops being documentation alone when its identifiers determine where infrastructure consumption is assigned.

This shift pressures platform teams to improve catalog quality. They must decide which annotations are mandatory, how identifiers are validated, and what happens when resources move between owners.

Data teams face a related change. They remain responsible for governed access and billing semantics, but they no longer need to own every integration pipeline connecting operational metadata with analytical records.

FinOps teams must adapt as well. Faster access does not eliminate their need to define allocation policies. It lets them apply those policies closer to the current operational model.

The payoff is therefore an altered division of labor. Platform engineering maintains a trustworthy ownership graph, the data platform supplies governed joins, and FinOps defines how consumption becomes accountability.

That arrangement is attractive because it removes waiting. It is demanding because errors in the ownership graph now flow directly into financial reporting.

The Mechanism Is Federation, Not a New Cost Database

Databricks is presenting federation as an alternative to copying operational catalog data into another cost-reporting store.

The core query begins with Backstage’s processed entity table. Backstage creates this representation after ingesting and normalizing catalog definitions from sources such as YAML files, plugins, or external systems.

The software catalog treats entities as metadata records describing components, systems, APIs, resources, groups, and users. Relations connect those entities into an ownership and dependency graph.

For the experiment, a Backstage resource contains the Lakebase project identifier in an annotation. The SQL query extracts that identifier from the entity’s JSON document.

It then joins the identifier to the billing table’s project metadata. Grouping by resource, project, and day produces an attribution result that humans can understand.

This mechanism has three significant properties.

First, the operational record stays operational. Backstage continues using PostgreSQL for its normal catalog activity rather than waiting for an analytical replica to accept updates.

Second, the billing record remains in the Databricks system table. The demonstration does not require a custom export before analysts can use it.

Third, the join happens through a governed query environment. Unity Catalog can control discovery and access across the participating objects.

Databricks calls this zero data movement. More precisely, the user does not build a separate pipeline to materialize another joined dataset before running the analysis.

Query execution still transfers requests and results across the relevant interfaces. Federation also depends on connectors, credentials, metadata, and performance controls.

That nuance does not erase the advantage. It clarifies where complexity goes after an ETL pipeline disappears.

Traditional pipelines express mappings in transformation code. The federated design expresses a critical mapping through an annotation placed on the Backstage entity.

The mapping becomes easier to see, but also easier to overlook. If the annotation is absent or incorrect, the query loses the connection between resource and owner.

A mature implementation would validate that annotation during catalog ingestion. It could reject malformed project identifiers or flag entities that reference missing Lakebase projects.

Teams would also need lifecycle rules. A deleted service, transferred application, or renamed project can create historical attribution questions that a current catalog snapshot cannot answer alone.

Point-in-time ownership becomes especially important for chargeback. Today’s owner should not automatically inherit consumption generated before a service transfer.

The demonstration’s daily grouping provides a starting point, but historical allocation requires durable ownership history. Teams might retain catalog changes, billing snapshots, or an effective-dated mapping table.

Access control adds another design decision. Engineers may need visibility into their own services without receiving unrestricted access to enterprise-wide billing records.

Unity Catalog can help define privileges, but each organization must establish the correct grain. Team-scoped views may be safer than granting broad access to the underlying system table.

Performance also deserves attention. The sample query is compact, yet production Backstage catalogs can contain many entities and large JSON records.

Repeated extraction from JSON may become inefficient at scale. Teams could expose selected annotations through a curated view while leaving the source catalog untouched.

That optimization would still preserve the broader model. The difference is that a governed semantic layer would sit above live operational data, rather than a disconnected export pipeline.

The databricks backstage architecture therefore does not abolish data engineering. It concentrates engineering work around metadata contracts, access policies, and query interfaces.

That is a more strategic location for the work, provided the organization accepts stricter catalog discipline.

The Real Opponent Is the ETL Handoff

The experiment challenges a pipeline-first integration model, not PostgreSQL competitors or other developer portals.

Teams traditionally separate online transaction processing from online analytical processing. Transactional systems prioritize frequent, low-latency writes, while analytical systems scan and aggregate much larger datasets.

That separation produced two operational domains. Application teams maintained databases such as PostgreSQL, while data teams copied selected records into warehouses or lakehouses.

The architecture was sensible because the systems had different storage layouts, scaling models, and failure concerns. Direct analytical queries could threaten application performance.

Modern cloud platforms increasingly separate storage, compute, and governance. That makes some data accessible through multiple workload-specific compute paths.

Lakebase sits inside this trend. Databricks positions it as managed PostgreSQL for transactional applications that also need proximity to lakehouse data.

The Backstage experiment uses that proximity to avoid a specialized ownership export. The primary alternative is not another Postgres vendor. It is the familiar process of extracting catalog records, transforming identifiers, and loading them into a reporting model.

ETL retains important advantages. A materialized dataset can provide predictable performance, durable snapshots, quality checks, and isolation from source schema changes.

It can also standardize data from several portals or cloud providers. A federated query against one Backstage database may not cover a company with multiple catalogs and billing environments.

The cost of ETL appears in latency and ownership. Someone must schedule the pipeline, monitor failures, update schemas, reconcile mappings, and respond when users distrust its output.

Those duties often fall between teams. Platform engineering understands the service catalog, data engineering owns the warehouse, and FinOps understands the allocation model.

A federation-first approach removes some copies and schedules. It makes the live source available but increases dependence on its uptime, schema, metadata quality, and query behavior.

This is the central tradeoff. ETL creates distance from the source and absorbs source instability. Federation offers freshness and fewer copies, but keeps consumers closer to operational change.

The right choice depends on the decision being supported. Interactive investigation benefits from current ownership data. Audited monthly chargeback requires stable history and reproducible rules.

A hybrid model is likely for larger organizations. Analysts can use a federated query to investigate recent usage, then publish approved allocation results into a durable reporting layer.

That model would not invalidate the demonstration. It would use federation to reduce exploratory friction while preserving materialized records for formal financial processes.

Other managed PostgreSQL platforms can also participate in federated architectures. AWS, Google Cloud, Microsoft, and independent database providers offer transactional systems with analytical integrations.

Backstage itself remains database-neutral. Its PostgreSQL support lets organizations choose infrastructure based on operational, regulatory, and commercial needs.

Databricks’ advantage in this experiment comes from platform proximity. Lakebase, system billing tables, federation, and Unity Catalog exist within one governed environment.

That convenience can also create concentration risk. A team that adopts the complete pattern becomes more dependent on Databricks identifiers, billing schemas, permissions, and query services.

The article’s competitive pressure therefore lands on fragmented internal stacks. Vendors and platform teams must explain why copying ownership metadata remains necessary when a governed query can reach the source.

However, Databricks must prove that the simpler diagram survives operational reality. That includes catalog scale, schema evolution, access boundaries, and month-end reporting demands.

What the One-Query Result Does Not Settle

A successful proof of concept does not guarantee trustworthy enterprise chargeback, because query correctness depends on organizational metadata outside the billing system.

The sample result shows that a Backstage resource can be matched to Lakebase usage. It does not show how much of a large organization’s infrastructure can be allocated this way.

Coverage is the first unanswered question. Teams need to know what percentage of relevant Lakebase projects have a valid Backstage entity and project annotation.

A result can be technically correct while remaining financially incomplete. Uncataloged projects simply vanish from an ownership-focused query unless analysts separately identify them.

Accuracy is the second question. A valid project ID can still point to the wrong service, an outdated owner, or a shared resource supporting several products.

Shared infrastructure complicates attribution because one project may serve several teams. A single Backstage annotation cannot express every proportional allocation rule.

Billing semantics create a third limitation. DBUs represent consumption, but a full cost view may need cloud infrastructure charges, credits, commitments, taxes, and organizational adjustments.

The FinOps team must decide whether the query supports showback, chargeback, anomaly investigation, or capacity planning. Those purposes require different levels of precision.

Historical ownership is another unresolved issue. The live catalog describes current state more naturally than past accountability.

A service transfer on April 15 should not necessarily rewrite who owned April 7 consumption. Reliable historical reports need time-aware ownership records.

Schema stability also matters. The query reaches into a JSON entity and depends on a specific annotation name. Teams must manage that field as a supported contract.

Backstage plugins, catalog processors, and organizational conventions can change entity shapes. Databricks can also evolve billing metadata as Lakebase develops.

Production users should test those contracts automatically. A validation job can identify missing annotations, unknown projects, duplicate assignments, and usage without an owner.

Security presents a different pressure. Backstage metadata may reveal internal systems and team structures, while billing data may expose sensitive consumption patterns.

Combining both datasets increases the information available from one query. That is useful for analysis, but it raises the impact of overly broad permissions.

Organizations need row or view-level boundaries that match their operating model. A service owner may need one team’s usage, while central FinOps needs enterprise coverage.

Operational dependency remains as well. Federated analysis relies on the source database, query service, identity configuration, and governance layer being available.

Databricks says Lakebase isolates compute for different workloads. Independent production evidence will still matter for latency, concurrency, failure recovery, and predictable analytical performance.

The original series also encountered an authentication complication. Lakebase expected a scoped OAuth credential rather than a classic Databricks personal access token.

For the proof of concept, the team refreshed a short-lived credential through a script. Production deployments need a supported rotation process that avoids storing refreshed tokens in unsafe locations.

Databricks Apps can create a PostgreSQL role for an application service principal when teams attach a Lakebase resource. That managed path is more suitable than an improvised local refresh loop.

The broader lesson is not that federation removes governance work. It makes governance visible at the moment a user asks a business question.

The databricks backstage pattern will succeed only when catalog completeness becomes an operational metric. Without that discipline, one-query FinOps can produce fast but partial answers.

Three Signals Will Show Whether the Pattern Holds

The next phase must measure adoption, allocation coverage, and production reliability rather than celebrate a compact SQL statement.

The first signal is a reusable implementation from Databricks, Thoughtworks, or the Backstage community. Teams should watch for maintained catalog processors, validation rules, dashboards, or templates supporting the pattern.

A supported package would strengthen the case that this architecture can move beyond a bespoke demonstration. It should define annotation schemas, credential handling, permissions, and deployment practices.

The absence of reusable components would weaken the claim. Every adopter would need to recreate the most failure-prone mappings and controls independently.

The second signal is allocation coverage in real catalogs. The useful metric is the percentage of Lakebase usage linked to a current, validated Backstage entity and owner.

High coverage across changing production environments would support federation as a practical FinOps input. Persistent unallocated usage would show that metadata maintenance remains the binding constraint.

Coverage should be paired with freshness and exception counts. Teams need to know how quickly new projects appear and how often mappings fail validation.

The third signal is production behavior under mixed workloads. Organizations should measure Backstage latency while analysts run federated billing queries against large catalogs.

Stable portal performance would reinforce Databricks’ separated-compute argument. Query interruptions, credential failures, or governance bottlenecks would favor materialized reporting layers for critical workflows.

The one-query case study offers a clear hypothesis: shared governance and isolated compute can reconnect operational ownership with analytical cost data.

Now teams need evidence about how that hypothesis behaves across multiple accounts, shared resources, service transfers, and formal financial controls.

For developers, the immediate action is simple. Treat catalog annotations as production data, validate infrastructure identifiers, and define ownership changes explicitly.

For platform leaders, ask whether the existing cost pipeline exists for a current requirement or an old architectural constraint. Federation may shorten investigations even when official reporting remains materialized.

For FinOps teams, test the databricks backstage query against known allocations before expanding it. The important question is not whether one query runs. It is whether its answer remains complete, explainable, and reproducible when the organization changes.