Lakebase Control DB Runbook

EvBricks can use Databricks Lakebase Postgres for Event Hub checkpoints and partition ownership. Use a Lakebase Autoscaling project with a native Postgres password role for the current implementation.

Do not use a raw Postgres server launched on Databricks compute for EvBricks control state. Databricks compute is not the managed durable Postgres service; use Lakebase for the Postgres-compatible endpoint.

1. Create Or Select Lakebase

In the Databricks workspace, create or select a Lakebase Postgres Autoscaling project in the same workspace region.

Select the branch, direct read-write endpoint, and default database from the Lakebase Connect dialog. The default Lakebase database is usually databricks_postgres, and the default schema is public.

Do not use the Lakebase pooler endpoint for the first EvBricks control DB configuration. EvBricks uses Postgres session advisory locks while creating the ownership table, so use the direct read-write endpoint copied from the Connect dialog.

2. Enable Password Connections

Native Postgres password connections are disabled by default for new Lakebase Autoscaling projects. Enable Password (Native Postgres roles) in the Lakebase project settings before using EvBricks.

CLI equivalent:

databricks postgres update-project projects/<project-id> spec.enable_pg_native_login \
  --json '{"spec": {"enable_pg_native_login": true}}'

If your selected compute endpoint has separate database connection settings, enable Password (Postgres roles) for that endpoint too.

3. Create Native Password Role

Create a native Postgres password role for EvBricks, for example evbricks_runtime. Use native password auth for the first working configuration because EvBricks currently uses static psycopg connection settings.

Lakebase OAuth roles require token rotation. The existing CONTROL_PG_AUTH_MODE=azure_token mode is for Azure Database for PostgreSQL Entra authentication, not Lakebase OAuth.

4. Prepare Schema

Connect as a role with admin privileges and run:

CREATE SCHEMA IF NOT EXISTS evbricks;

GRANT CONNECT, CREATE ON DATABASE databricks_postgres TO evbricks_runtime;
GRANT USAGE, CREATE ON SCHEMA evbricks TO evbricks_runtime;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA evbricks TO evbricks_runtime;

ALTER DEFAULT PRIVILEGES IN SCHEMA evbricks
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO evbricks_runtime;

EvBricks creates or verifies evbricks.ingestion_status on startup. The Event Hubs SDK checkpoint store creates evbricks.ingestion_status_ownership when partition ownership is claimed.

5. Configure EvBricks

Use this control section in config/evbricks.toml:

[control]
backend = "postgres"
target_db = "databricks_postgres"
target_schema = "evbricks"
target_table = "ingestion_status"

Use these .env values from the Lakebase Connect dialog:

CONTROL_PG_HOST=<lakebase-direct-read-write-endpoint-host>
CONTROL_PG_PORT=5432
CONTROL_PG_USER=evbricks_runtime
CONTROL_PG_PASSWORD=<lakebase-native-postgres-password>
CONTROL_PG_SSLMODE=require
CONTROL_PG_AUTH_MODE=password

Keep the existing Databricks Zerobus variables in the same .env file:

DATABRICKS_WORKSPACE_URL=https://<workspace-host>
ZEROBUS_SERVER_ENDPOINT=https://<workspace-id>.zerobus.<region>.azuredatabricks.net
DATABRICKS_CLIENT_ID=<service-principal-application-id>
DATABRICKS_CLIENT_SECRET=<service-principal-secret>

6. Verify Control Connectivity

uv run evbricks validate-config \
  --config-file config/evbricks.toml \
  --env-file .env \
  --require-runtime-secrets

Optionally verify the Lakebase role with the Databricks CLI and local psql:

databricks psql projects/<project-id>/branches/<branch-id>/endpoints/<endpoint-id> \
  -- -c "SELECT current_database(), current_user"

After a live EvBricks run starts, verify checkpoint/control rows:

SELECT
  eventhub_namespace,
  eventhub,
  target_db,
  target_schema,
  target_table,
  partition_id,
  waterlevel,
  ts_inserted
FROM evbricks.ingestion_status
ORDER BY partition_id;

SELECT
  fully_qualified_namespace,
  eventhub_name,
  consumer_group,
  owner_id,
  partition_id,
  last_modified_time
FROM evbricks.ingestion_status_ownership
ORDER BY partition_id;