Skip to content

Local Smoke Runbook

1. Configure Local Files

cp config/evbricks.example.toml config/evbricks.toml
cp .env.example .env

Edit local files with real non-secret names and secret values. Do not commit either file.

2. Validate Shape

uv run evbricks validate-config --config-file config/evbricks.toml --env-file .env
uv run evbricks run --config-file config/evbricks.toml --env-file .env --dry-run

These commands must not open Event Hub or Zerobus clients.

3. Live Smoke Prerequisites

  • Workspace and target table are in the same Zerobus-supported region.
  • Target table is a Unity Catalog managed Delta table and is not backed by unsupported default/private-endpoint storage.
  • Target table schema matches ../../sql/create_eventhub_events.sql.
  • Target managed Delta table exists at the configured target FQN from config/evbricks.toml.
  • Databricks service principal exists.
  • Service principal has USE CATALOG, USE SCHEMA, MODIFY, and SELECT on the target.
  • Lakebase Postgres control DB is configured with a native password role. See Lakebase control DB runbook.
  • config/evbricks.toml uses databricks_postgres.evbricks.ingestion_status for control state unless a different Lakebase database/schema was intentionally created.
  • Event Hub identity has Azure Event Hubs Data Receiver.
  • Sender identity has Azure Event Hubs Data Sender if using the included sender.

4. Live Acceptance

Before sending Event Hub messages, open a Zerobus stream against the target table. If stream creation fails, stop here and do not send a batch.

Start EvBricks first and confirm it opens the Zerobus stream without errors:

uv run evbricks run --config-file config/evbricks.toml --env-file .env

After EvBricks is running and the destination stream is open, send a small known batch to Event Hub. Stop the process after the expected rows have landed. Query Databricks for the expected sequence IDs and Event Hub metadata:

SELECT
  COUNT(*) AS rows_arrived,
  sort_array(collect_set(CAST(get_json_object(body, '$.sequence_id') AS BIGINT))) AS sequence_ids,
  collect_set(eventhub_namespace) AS namespaces,
  collect_set(eventhub_name) AS event_hubs,
  collect_set(consumer_group) AS consumer_groups,
  count(DISTINCT concat(partition_id, ':', offset)) AS distinct_eventhub_offsets,
  count(DISTINCT sequence_number) AS distinct_eventhub_sequence_numbers,
  sum(CASE WHEN sequence_number IS NULL THEN 1 ELSE 0 END) AS null_eventhub_sequence_numbers,
  count(*) - count(DISTINCT concat(
    eventhub_namespace,
    '|',
    eventhub_name,
    '|',
    consumer_group,
    '|',
    partition_id,
    '|',
    offset,
    '|',
    sequence_number
  )) AS duplicate_identity_rows
FROM <catalog>.<schema>.eventhub_events
WHERE body LIKE concat('%', '<run-id>', '%');

Expected result:

  • rows_arrived equals the number of sent messages.
  • sequence_ids contains every sent sequence_id.
  • namespaces, event_hubs, and consumer_groups match the configured Event Hub source.
  • distinct_eventhub_offsets equals rows_arrived.
  • distinct_eventhub_sequence_numbers equals rows_arrived.
  • null_eventhub_sequence_numbers equals 0.
  • duplicate_identity_rows equals 0.
  • Re-running after shutdown resumes from the latest Postgres checkpoint and does not reprocess checkpointed messages.

Verify Lakebase checkpoint/control state:

SELECT partition_id, waterlevel, ts_inserted
FROM evbricks.ingestion_status
ORDER BY partition_id;

SELECT partition_id, owner_id, last_modified_time
FROM evbricks.ingestion_status_ownership
ORDER BY partition_id;

5. Local Verification Commands

uv run pytest tests/unit -q
uv run ruff check .
uv run evbricks --help
uv run evbricks validate-config --config-file config/evbricks.example.toml
uv run evbricks run --config-file config/evbricks.example.toml --dry-run

6. Live Smoke Result Template

Record live smoke evidence outside Git, or in a sanitized issue/PR note:

workspace: <redacted>
region: <region>
target: <catalog>.<schema>.eventhub_events
Event Hub: <namespace>/<eventhub>
run_id: <run-id>
sent messages: <count>
rows_arrived: <count>
sequence_ids: <sent-sequence-range>
duplicate_identity_rows: 0
checkpoint summary: <checkpoint-row-count-and-waterlevel>