Skip to content

Migration From EvSnow

EvBricks keeps EvSnow's reliable Event Hub side and replaces Snowflake ingestion with Databricks Zerobus.

Scope

  • Keep Event Hub production, receiving, batching, and checkpoint-after-success behavior.
  • Replace Snowflake streaming clients, Snowflake target config, and Snowflake control-table paths.
  • Use Databricks Zerobus only for writes into Databricks managed Delta tables.
  • Use Databricks Lakebase Postgres for Event Hub checkpoint and partition ownership.

Feasibility Decision

This is feasible as an Event Hubs to Zerobus bridge. Zerobus is a push ingestion API, not an Event Hubs source connector, so EvBricks still needs an Event Hub consumer process. The destination write path must return success only after Zerobus has acknowledged durability; only then can Event Hub checkpoints advance.

Live acceptance requires explicit Unity Catalog managed storage in the same supported Azure region as the Databricks workspace. Do not use default workspace storage or private-endpoint-only storage for the first Zerobus smoke.

First Target Shape

Source: - Azure Event Hubs SDK consumer - Existing EvSnow message envelope fields - Batch by Event Hub partition before writing

Destination: - Databricks Zerobus SDK in JSON mode - Unity Catalog managed Delta target table - Non-default managed storage backed by ADLS Gen2 in the same supported region as the workspace - Stable EvBricks source ID derived from Event Hub, environment, region, and client ID for logs and stats only; this is not a Zerobus SDK stream-name argument

Control: - Lakebase Postgres checkpoint table for Event Hub checkpoints and ownership - No Snowflake control state

Target Table Contract

The first target table is the configured Unity Catalog table, for example <catalog>.<schema>.eventhub_events, created from ../sql/create_eventhub_events.sql after adjusting the catalog and schema for the supported non-default managed-storage target. It mirrors the keys emitted by EventHubMessage.to_dict() and keeps timestamps and JSON metadata as strings for the first smoke test. Zerobus does not auto-evolve target schemas, so writer changes must update the DDL and tests together.

The accepted target must be a Unity Catalog managed Delta table in a supported region and backed by non-default managed storage that Zerobus accepts.

Live Acceptance Evidence

For a deployment to pass live acceptance:

  • Direct Zerobus smoke inserts a row into the configured target table.
  • EvBricks opened Zerobus before Event Hub messages were sent.
  • EvSnow sender emits a small batch with a unique run ID.
  • Databricks query returns the expected row count, namespace, hub, consumer group, and distinct Event Hub offsets.
  • Payload sequence IDs exactly match the sent range.
  • Event Hub sequence_number is present for all rows and distinct across the batch.
  • Duplicate Event Hub identity rows were 0.
  • Lakebase checkpoint state advanced and partition ownership rows existed.

Test Matrix

Area Test Acceptance
Config TOML loads and applies Event Hub defaults Local smoke uses azure_cli when the example says so
CLI validate/dry-run without secrets No live clients are opened and warnings are visible
Runtime validation non-dry-run requires Databricks and Postgres settings Placeholder or absent secrets fail before live clients open
Writer success mocked Zerobus batch write waits for durability returns true only after final offset/flush succeeds
Writer failure mocked Zerobus raises retryable/non-retryable errors returns false or raises without checkpoint advancement
Checkpoints Event Hub checkpoint advances only after writer success failed batch remains uncheckpointed
Restart restart after checkpoint resumes from persisted Event Hub checkpoint
Dedupe duplicate Event Hub metadata exists in target rows (namespace, hub, group, partition, offset, sequence) can dedupe
Live smoke send N known messages Databricks target contains N matching rows
Storage gate open Zerobus stream on target table succeeds before Event Hub messages are sent
Azure full workspace send a known EvSnow batch through EvBricks Databricks target contains exactly the sent rows and no duplicates

Acceptance Criteria

  • evbricks validate-config --config-file config/evbricks.example.toml does not touch live services.
  • evbricks run --config-file config/evbricks.example.toml --dry-run prints source and target mappings.
  • Unit tests cover config, message shaping, and writer contract before any live smoke.
  • Zerobus stream creation succeeds for the target table before any Event Hub messages are sent.
  • Live smoke sends known Event Hub messages and verifies matching Databricks rows.