Databricks Health Check: Spend, Governance, Delta Tables and Jobs Reviewed

What a Databricks health check reads, and where the evidence comes from

A Databricks health check works from evidence your own platform already records. Databricks publishes system tables for billing, compute, access, jobs, pipelines, query history and predictive optimization, so most findings can be traced to rows in a table your engineers can run a query against, not to anyone's recollection.

Access to system tables is granted by an admin, with USE CATALOG, USE SCHEMA and SELECT on the system schemas. That is why delivery is guided by your engineers: they run the read-only queries and exports, we read the results.

For a service-level example of what a usage review finds, see the DBU cost governance use case

Delta table health: small files, OPTIMIZE, VACUUM and clustering

Delta table health is mostly file layout. A table made of many small files makes every read open more files than the row count justifies, and a table that is never vacuumed keeps storage for data nobody queries. The check starts with DESCRIBE DETAIL, which returns the file count, the size, the partition columns and the clustering columns of each table.

For a service-level example of fixing this layout problem, see the Delta small-file remediation use case

Unity Catalog governance review

The governance review compares your Unity Catalog setup with the practices Databricks itself documents, and reports where the estate differs and what the difference costs you in risk or admin effort.

• Groups, not users — Databricks says to set up groups and avoid direct grants to users whenever possible. We count direct user grants and the objects they sit on.
• Ownership — Object ownership should sit with groups, especially in production. Objects owned by people who have left, or by their creators by default, are listed.
• Service principals run production — Databricks recommends running jobs as service principals and reserving direct MODIFY access on production tables for them. We check who can write to production tables today.
• External locations — An external location pairs a storage credential with a storage path. Creating them should be limited to administrators and trusted engineers, and broad READ FILES or WRITE FILES grants should be avoided.
• Managed versus external tables — Databricks recommends managed tables for most uses and all new tables. We list external tables and the reason each one exists, since the answer decides whether predictive optimization applies.
• Catalog layout and binding — Catalogs should reflect environments, teams or business units, and can be bound to specific workspaces when isolation is needed. One metastore serves each region, so workspaces in a region share it.

Lineage and audit data complete the picture. The system.access.table_lineage and system.access.audit tables show which tables are read, by whom, and which grants and changes happened, so the review rests on use as well as on configuration.

Greenfield setup of the same model is covered under Databricks implementation

Security posture: network, secrets, keys and audit

The security review looks at configuration, not at your code. Databricks publishes an open-source Security Analysis Tool that compares account and workspace settings with its security best practices, and we use findings like that as an input, then review them in context rather than repeating a scan.

Networking and key-management features differ by cloud provider, so we review against the documentation for the cloud you run on. The review reports gaps against your own policy. It is not a compliance certification.

Cluster, job and pipeline reliability

Configuration problems and reliability problems overlap. A job on an unsupported runtime fails differently from one that runs out of memory, and a job with no retry policy fails differently from one that retries a non-idempotent task. We read both from job run history and from the settings behind it.

• Runtime support lifecycle — Databricks Runtime LTS releases are supported for three years. After end of support, workloads get no Databricks support, fixes are not backported, and the version can no longer be picked in the UI. We list the runtimes in use and their position in that lifecycle.
• Retries and timeouts — For most job configurations the default is that tasks are not retried on failure. If you set both a timeout and retries, the timeout applies to each retry. We check which critical tasks have neither.
• Repair runs — A repair run re-runs only the failed tasks and their dependents. Databricks warns that tasks are not made idempotent, so a task that wrote part of its output before failing can duplicate data when repaired. We look for tasks where that risk is real.
• Notifications — Jobs can notify on start, success, failure, a duration threshold or a streaming backlog threshold, through email, Slack, Microsoft Teams, PagerDuty or any webhook-based service. We check what pages a person and what only writes a log.
• Data quality expectations — Expectations in Lakeflow pipelines can warn, drop invalid rows or fail the update. The default is to warn and keep the rows, so we check where a failure would be preferable.
• Failure history — system.lakeflow.job_run_timeline and job_task_run_timeline carry result states and termination codes, so recurring failures can be counted and grouped by cause.

For the spend side of cluster and job configuration, see Databricks cost optimization

What the Databricks health check report contains

The report is written to be read by the engineer who runs the platform and the manager who funds the fixes. This is the outline every report follows. It is a structure, not a client document, and it carries no customer data.

A report with a hundred undifferentiated observations has not done the hard part, which is ranking them. Ours puts the few findings that matter first and the rest in the appendix.

The practice is led by a former Databricks professional-services resident solution architect.

Where this health check stops

• Remote and read-only — We work from exports and a screen share, guided by your engineers. We do not log in to your workspaces or touch your code.
• No savings promised in advance — The health check measures spend by workload and ranks drivers. We do not quote a percentage before seeing your usage.
• Not a compliance audit — We report gaps against Databricks' documented practice and your own policy. We do not certify that you meet a regulation or standard.
• Remediation is separate — The report stands alone. If you want us to carry out the fixes, that is scoped and quoted as its own piece of work.
• AceMQ is independent of Databricks — We do not resell Databricks, and licensing and contract terms are between you and Databricks or your cloud marketplace.

If findings point to spend, start with Databricks cost optimization

If findings point to a platform move, see Databricks migration services

For production cover once fixed, see Databricks support