Hotrod
Foglight for Databricks
Workspace-wide visibility into the lakehouse
Description
Databricks is the unified data and AI platform built on the lakehouse architecture, combining the scale of a data lake with the reliability and performance of a data warehouse. Teams run interactive clusters, automated jobs, SQL warehouses, and ML workloads inside workspaces that can sprawl quickly across an organization.
Foglight for Databricks is an SMA cartridge that polls the Databricks REST APIs to bring that activity into Foglight, giving platform owners a single place to watch the health, cost drivers, and reliability of their workspaces.
Business Challenge
Databricks usage is elastic by design - clusters spin up and down, jobs succeed and fail, and consumption can climb without anyone watching. The native UI is excellent for a single workspace in the moment, but provides little in the way of cross-workspace history, proactive alerting, or correlation with the rest of the data estate.
By treating the workspace as a first-class monitored object, Foglight for Databricks lets you trend job and cluster behavior over time and alert on the conditions that matter to your team.
Key Features
The cartridge is workspace-keyed: the DatabricksWorkspace is the top-level topology object. The agent authenticates to the Databricks REST API and collects the platform telemetry that surfaces in clean, responsive dashboards.
As with every Foglight solution, it runs as a native Foglight process on the Foglight Agent Manager (FglAM), independent of the monitored workspace, and leverages the Foglight Rules Engine for threshold and anomaly alerting.
Dashboards
Screenshots
Ready to see Databricks in action?
Request a free trial or talk to our engineers about your environment.