Skip to content

What EvBricks is

EvBricks is a streaming ingestion bridge. It reads Azure Event Hubs events and writes rows to Databricks through the Zerobus Ingest connector.

EvBricks is not a batch ETL system. It is a long-running consumer that keeps Event Hub offsets, Databricks write acknowledgements, and checkpoint state in a single success path.

What lives where

flowchart TB
  subgraph azure["Azure"]
    namespace["Event Hubs namespace"]
    hub["Event hub"]
    cg["Consumer group"]
    parts["Partitions, offsets, sequence numbers"]
    senderRole["Azure Event Hubs Data Sender"]
    receiverRole["Azure Event Hubs Data Receiver"]
  end

  subgraph runtime["EvBricks runtime host"]
    cli["evbricks CLI"]
    cfg["config/evbricks.toml"]
    env[".env runtime secrets"]
    consumer["EventHubConsumerClient"]
    mapper["EventHubMessage.to_dict()"]
    writer["Zerobus SDK client"]
  end

  subgraph databricks["Databricks"]
    workspace["Databricks workspace"]
    endpoint["Zerobus Ingest endpoint"]
    sp["Databricks service principal"]
    uc["Unity Catalog catalog and schema"]
    table["Managed Delta table"]
    lakebase["Lakebase Postgres"]
    checkpoints["Checkpoint and ownership tables"]
  end

  namespace --> hub
  hub --> parts
  cg --> consumer
  parts --> consumer
  receiverRole --> consumer
  senderRole --> hub
  cli --> cfg
  cli --> env
  consumer --> mapper
  mapper --> writer
  writer --> endpoint
  sp --> endpoint
  endpoint --> table
  table --> uc
  consumer --> checkpoints
  checkpoints --> lakebase
  workspace --> endpoint
  workspace --> uc
  workspace --> lakebase

Azure side

Azure owns the event stream.

  • An Event Hubs namespace is the management container.
  • An event hub is the append-only stream that receives events.
  • Partitions provide parallel ordered streams. EvBricks stores the Event Hub partition_id, offset, and sequence_number in the target table.
  • A consumer group gives EvBricks an independent view of the event stream.
  • The sender identity needs permission to send events. The EvBricks runtime identity needs permission to receive events.

The included sender is only a test utility. Production producers can be any application that sends valid events to the event hub.

Databricks side

Databricks owns the destination and control state.

  • The Databricks workspace provides the workspace URL and workspace identity boundary.
  • The Zerobus Ingest endpoint receives write requests for the workspace and region.
  • A Databricks service principal authenticates the Zerobus write path.
  • Unity Catalog contains the catalog, schema, and managed Delta table that stores Event Hub rows.
  • Lakebase Postgres stores EvBricks checkpoint and ownership state.

The target table is created before EvBricks runs. Zerobus does not decide the schema for EvBricks; the table must already match the row shape documented in sql/create_eventhub_events.sql.

Runtime side

The EvBricks process joins the two clouds.

  • config/evbricks.toml defines non-secret pipeline shape: Event Hub keys, Databricks target keys, table names, batch size, and mapping.
  • .env or process environment variables define runtime secrets and local credentials.
  • EventHubConsumerClient receives events from Azure Event Hubs.
  • EventHubMessage.to_dict() converts each event to the target row shape.
  • DatabricksZerobusIngestionClient writes JSON records through Zerobus and waits for the returned offset when wait_for_offsets = true.
  • Checkpoint updates happen after the write succeeds.

Batch success path

sequenceDiagram
  participant P as Producer application
  participant EH as Azure Event Hubs
  participant EB as EvBricks
  participant ZB as Zerobus Ingest
  participant DT as Managed Delta table
  participant PG as Lakebase Postgres

  P->>EH: Send events
  EB->>EH: Receive events by consumer group
  EB->>EB: Build batch and preserve metadata
  EB->>ZB: ingest_records_offset(JSON rows)
  ZB->>DT: Write rows
  ZB-->>EB: Return offset acknowledgement
  EB->>ZB: wait_for_offset(offset)
  EB->>PG: Update checkpoint after success

The important invariant is: EvBricks checkpoints after Databricks write acknowledgement, not after Event Hubs receive.

Row shape

Each Databricks row contains:

  • Event Hub identity: namespace, event hub name, consumer group, partition, offset, and sequence number.
  • Event timestamps and content type.
  • Event body as a string.
  • Event properties and system properties as serialized JSON strings.
  • Ingestion timestamp generated by EvBricks.

The first implementation writes JSON records. Protocol Buffers and Arrow Flight are Databricks Zerobus options, but EvBricks currently uses JSON because it matches the flexible Event Hub smoke-test payloads.

Why Lakebase Postgres is used

Checkpoint state is operational state, not analytical data. Lakebase Postgres is a fully managed Postgres database integrated into Databricks, which makes it a natural control-plane store for EvBricks when the destination is also Databricks.

EvBricks uses Postgres tables for:

  • Last processed Event Hub offset and sequence number per partition.
  • Partition ownership state used by the Event Hubs SDK checkpoint store.

Design boundaries

EvBricks intentionally keeps these boundaries clear:

  • Azure Event Hubs is the source stream.
  • Databricks Zerobus is the write transport.
  • Unity Catalog managed Delta is the durable analytical destination.
  • Lakebase Postgres is the durable checkpoint/control backend.
  • The local runtime host owns only process execution and ignored local config.