Configuration

EvBricks reads configuration from TOML, .env, and process environment variables.

The usual local files are:

config/evbricks.toml for pipeline shape.
.env for secrets and live endpoints.

Environment variables override only the top-level and runtime fields documented on this page. Nested Event Hub and Databricks target shape should stay in TOML.

Top-level TOML fields

Field	Type	Required	Default	Description
`eventhub_namespace`	string	yes	none	Top-level Azure Event Hubs namespace. Must end with `.servicebus.windows.net`. Set each `[event_hubs.<KEY>].namespace` too.
`environment`	string	no	`development`	Environment segment used in source IDs and logs.
`region`	string	no	`local` in TOML model, `default` in runtime model	Region segment used in source IDs and run context.
`client_id`	string	no	host name	Stable EvBricks runtime identifier used in source IDs.
`max_concurrent_mappings`	integer	no	`10`	Reserved for limiting concurrent mapping work.
`health_check_interval_seconds`	integer	no	`60`	Reserved health-check interval.
`enable_detailed_logging`	boolean	no	`false`	Enables more detailed logging where implemented.
`metrics_collection_enabled`	boolean	no	`true`	Enables metrics collection where implemented.

`[control]`

The control section configures checkpoint table names. EvBricks currently uses Postgres for this backend.

Field	Type	Required	Default	Description
`backend`	string	no	`postgres`	Must be `postgres`.
`target_db`	string	no	`databricks_postgres`	Postgres database that contains the control schema.
`target_schema`	string	no	`evbricks`	Postgres schema for control tables.
`target_table`	string	no	`ingestion_status`	Checkpoint table name.

`[eventhub_defaults]`

Defaults apply to every entry under [event_hubs.*] unless the individual hub sets the same field.

Field	Type	Default	Description
`connection_string`	string	unset	Optional Event Hubs connection string. Prefer Entra ID for normal use.
`credential_mode`	`default` or `azure_cli`	`default`	Use `azure_cli` for local smoke tests. Use `default` for production-capable `DefaultAzureCredential`.
`managed_identity_client_id`	string	unset	User-assigned managed identity client ID for `DefaultAzureCredential`.
`max_batch_size`	integer	`1000`	Accepted by the model for compatibility; not currently passed to the runtime receive loop.
`max_wait_time`	integer	`60`	Receive idle wait in seconds. `0` means wait until an event arrives.
`prefetch_count`	integer	`300`	Event Hubs SDK prefetch count.
`retry_total`	integer	`3`	Event Hubs SDK retry attempts.
`retry_backoff_factor`	number	`0.8`	Retry backoff factor.
`retry_backoff_max`	number	`120`	Maximum retry backoff.
`retry_mode`	`exponential` or `fixed`	`exponential`	Event Hubs SDK retry mode.
`load_balancing_interval`	number	`30`	Partition ownership load-balancing interval.
`partition_ownership_expiration_interval`	number	unset	If set, must be at least `6 * load_balancing_interval`.
`load_balancing_strategy`	`greedy` or `balanced`	`greedy`	Event Hubs SDK partition load-balancing strategy.
`track_last_enqueued_event_properties`	boolean	`false`	Enables last-enqueued event tracking in the Event Hubs SDK.
`checkpoint_interval_seconds`	integer	`300`	Reserved checkpoint timing setting; not currently passed to the runtime receive loop.
`max_message_batch_size`	integer	`1000`	Reserved maximum processing batch setting; not currently passed to the runtime receive loop.
`batch_timeout_seconds`	integer	`300`	Accepted by the model; the runtime currently uses the consumer constructor default.
`starting_position_on_no_checkpoint`	`-1` or `@latest`	`-1`	Start from beginning or latest position when no checkpoint exists.

`[event_hubs.<KEY>]`

Each Event Hub key must be referenced by a mapping.

Field	Type	Required	Default	Description
`name`	string	yes	none	Event hub name.
`namespace`	string	yes	none	Event Hubs namespace, for example `eventhub1.servicebus.windows.net`. This is required even when the top-level `eventhub_namespace` has the same value.
`consumer_group`	string	yes	none	Event Hubs consumer group name.
`connection_string`	string	no	unset	Optional per-hub connection string.

All fields from [eventhub_defaults] can also be set on an individual Event Hub.

`[databricks_configs.<KEY>]`

Each Databricks key describes one Zerobus target table.

Field	Type	Required	Default	Description
`catalog`	string	yes	none	Unity Catalog catalog name. ASCII letters, digits, and underscores only.
`schema_name`	string	yes	none	Unity Catalog schema name. ASCII letters, digits, and underscores only.
`table_name`	string	yes	none	Managed Delta table name. ASCII letters, digits, and underscores only.
`batch_size`	integer	no	`1000`	Number of Event Hub messages per EvBricks processing batch.
`record_type`	`json`	no	`json`	EvBricks currently writes JSON records through Zerobus.
`max_inflight_records`	integer	no	`10000`	Zerobus stream option.
`recovery`	boolean	no	`true`	Zerobus stream recovery option.
`recovery_timeout_ms`	integer	no	`15000`	Zerobus recovery timeout.
`recovery_backoff_ms`	integer	no	`2000`	Zerobus recovery backoff.
`recovery_retries`	integer	no	`3`	Zerobus recovery retry count.
`flush_timeout_ms`	integer	no	`300000`	Zerobus flush timeout.
`server_lack_of_ack_timeout_ms`	integer	no	`60000`	Zerobus acknowledgement timeout.
`wait_for_offsets`	boolean	no	`true`	Wait for the returned Zerobus offset before checkpointing.

The full target table name is:

<catalog>.<schema_name>.<table_name>

`[[mappings]]`

Mappings connect an Event Hub key to a Databricks target key.

Field	Type	Required	Default	Description
`event_hub_key`	string	yes	none	Key under `[event_hubs]`. Must be uppercase with underscores.
`databricks_key`	string	yes	none	Key under `[databricks_configs]`. Must be uppercase with underscores.
`source_id_pattern`	string	no	`{event_hub}-{env}-{region}-{client_id}`	Format string for Zerobus source IDs.

Allowed source_id_pattern placeholders:

{event_hub}
{env}
{region}
{client_id}

EvBricks sanitizes source ID segments to ASCII letters, digits, underscores, and hyphens before formatting.

Databricks runtime environment

Set these in .env or the process environment:

Variable	Required	Description
`DATABRICKS_WORKSPACE_URL`	yes	Databricks workspace URL, without path fragments.
`ZEROBUS_SERVER_ENDPOINT`	yes	Zerobus Ingest endpoint for the workspace and region.
`DATABRICKS_CLIENT_ID`	yes	Databricks service principal application/client ID.
`DATABRICKS_CLIENT_SECRET`	yes	Databricks service principal secret.

Optional TOML runtime sections

The TOML loader accepts these sections, but keep secrets in .env for normal local and shared repo workflows.

`[databricks_connection]`

Field	Description
`workspace_url`	Same value as `DATABRICKS_WORKSPACE_URL`.
`zerobus_endpoint`	Same value as `ZEROBUS_SERVER_ENDPOINT`.
`client_id`	Same value as `DATABRICKS_CLIENT_ID`.
`client_secret`	Same value as `DATABRICKS_CLIENT_SECRET`; avoid storing it in TOML.

Environment variables with the Databricks names above override this TOML section when both are present.

`[control.postgres]`

Field	Description
`host`	Same value as `CONTROL_PG_HOST`.
`port`	Same value as `CONTROL_PG_PORT`.
`user`	Same value as `CONTROL_PG_USER`.
`password`	Same value as `CONTROL_PG_PASSWORD`; avoid storing it in TOML.
`sslmode`	Same value as `CONTROL_PG_SSLMODE`.
`auth_mode`	Same value as `CONTROL_PG_AUTH_MODE`.

If this section is absent, EvBricks resolves Postgres settings from CONTROL_PG_* environment variables.

`[logfire]`

Field	Description
`enabled`	Same value as `LOGFIRE_ENABLED`.
`token`	Same value as `LOGFIRE_TOKEN`; avoid storing it in TOML.
`service_name`	Same value as `LOGFIRE_SERVICE_NAME`.
`environment`	Same value as `LOGFIRE_ENVIRONMENT`.
`send_to_logfire`	Same value as `LOGFIRE_SEND_TO_LOGFIRE`.
`console_logging`	Same value as `LOGFIRE_CONSOLE_LOGGING`.
`log_level`	Same value as `LOGFIRE_LOG_LEVEL`.

Lakebase Postgres environment

Set these when [control].backend = "postgres":

Variable	Required	Default	Description
`CONTROL_PG_HOST`	yes	none	Lakebase Postgres host.
`CONTROL_PG_PORT`	no	`5432`	Postgres port.
`CONTROL_PG_USER`	yes	none	Postgres user.
`CONTROL_PG_PASSWORD`	yes for password auth	none	Postgres password.
`CONTROL_PG_SSLMODE`	no	`require`	Postgres SSL mode.
`CONTROL_PG_AUTH_MODE`	no	`password`	`password` for Lakebase native roles; `azure_token` only for Azure Database for PostgreSQL Entra authentication.

Logfire environment

Logfire is optional.

Variable	Default	Description
`LOGFIRE_ENABLED`	`false`	Enable Logfire instrumentation.
`LOGFIRE_TOKEN`	unset	Required when Logfire is enabled and cloud sending is enabled.
`LOGFIRE_SERVICE_NAME`	`evbricks`	Service name.
`LOGFIRE_ENVIRONMENT`	`development`	Environment tag.
`LOGFIRE_SEND_TO_LOGFIRE`	`true`	Send to Logfire cloud when enabled.
`LOGFIRE_CONSOLE_LOGGING`	`true`	Keep Rich console logging.
`LOGFIRE_LOG_LEVEL`	`INFO`	One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`.

Top-level environment overrides

Variable	Description
`EVENTHUB_NAMESPACE`	Overrides the top-level `eventhub_namespace` only. It does not fill or override `[event_hubs.<KEY>].namespace`.
`ENVIRONMENT`	Overrides `environment`.
`REGION`	Overrides `region`.
`EVBRICKS_CLIENT_ID`	Overrides `client_id`.
`TARGET_DB`	Overrides `[control].target_db`.
`TARGET_SCHEMA`	Overrides `[control].target_schema`.
`TARGET_TABLE`	Overrides `[control].target_table`.
`CONTROL_TABLE_BACKEND`	Overrides `[control].backend`. Must resolve to `postgres`.
`MAX_CONCURRENT_MAPPINGS`	Overrides `max_concurrent_mappings`.
`HEALTH_CHECK_INTERVAL_SECONDS`	Overrides `health_check_interval_seconds`.
`ENABLE_DETAILED_LOGGING`	Overrides `enable_detailed_logging`.
`METRICS_COLLECTION_ENABLED`	Overrides `metrics_collection_enabled`.

EvBricks does not currently support environment-variable overrides for nested [event_hubs.<KEY>] or [databricks_configs.<KEY>] pipeline shape. Prefer TOML for those values.

The included sender is separate from the EvBricks runtime and can still read AZURE_EVENTHUB_CONNECTION_STRING, EVENTHUB_NAMESPACE, and EVENTHUBNAME_1 from .env when you run tools/eventhub_sender/main.py.

Validation commands

Validate static shape:

uv run evbricks validate-config --config-file config/evbricks.example.toml

Validate live runtime settings:

uv run evbricks validate-config \
  --config-file config/evbricks.toml \
  --env-file .env \
  --require-runtime-secrets

Preview the processing plan without opening live clients:

uv run evbricks run \
  --config-file config/evbricks.toml \
  --env-file .env \
  --dry-run

Secret handling

Do not commit:

.env
config/evbricks.toml
client secrets
Postgres passwords
Event Hubs connection strings
private keys, certificates, token files, or credential JSON

Use the committed examples for reusable shape:

.env.example
config/evbricks.example.toml
schemas/evbricks.schema.json