Skip to content
Coordinator DB
View as Markdown
View as Markdown

avro_confluent ​

Description ​

Consecutive Avro records prefixed with a zero byte and a big-endian registry schema ID. Configure avro_confluent_registry_url; output also needs avro_confluent_subject. This is a schema-registry format, rather than an Avro object container.

Input and output. Select it with FORMAT avro_confluent. Names ignore ASCII case; use the snake_case name shown here.

Every record carries a schema identifier rather than an embedded object-container schema. Input resolves that identifier through the configured registry. Output registers or obtains a schema under the configured subject.

Supply a reachable registry URL on the statement. Output also needs a writable subject. These are external-service requirements; ordinary avro files use a different envelope.

text
SELECT expressions FORMAT avro_confluent;
INSERT INTO table [(columns)] [SETTINGS ...] FORMAT avro_confluent
<data>

Example usage ​

Use the native client's --query mode to preserve binary output bytes. The server must be running, the executable available as ./coordinator, and database data selected.

Set avro_confluent_registry_url to your running schema registry and avro_confluent_subject to a writable subject before export. A registry-dependent command requires that external service.

sql
SELECT name,is_input,is_output FROM system.formats WHERE name='avro_confluent' FORMAT csv;
text
"avro_confluent",1,1

Format settings ​

Use a statement SETTINGS clause; these options are not persisted by SET.

Setting Type Default Meaning
avro_confluent_connect_timeout UInt64 1 Connection and TLS handshake timeout in seconds; positive values are capped at 599.
avro_confluent_receive_timeout UInt64 1 Registry response receive timeout in seconds; positive values are capped at 599.
avro_confluent_registry_url String '' HTTP or HTTPS schema registry base URL; URL-encoded credentials select Basic authentication.
avro_confluent_retries UInt64 5 Retry transient registry failures this many times; zero disables retries and values above 20 are capped at 20.
avro_confluent_retry_delay_ms UInt64 100 Initial retry delay in milliseconds, capped at 60000; subsequent delays double up to 10000 milliseconds.
avro_confluent_send_timeout UInt64 1 Registry request send timeout in seconds; positive values are capped at 599.
avro_confluent_subject String '' Subject under which an output record schema is registered; required for avro_confluent output.
avro_missing_as_default Bool 0 Give target fields absent from the input schema their SQL DEFAULT or native type default; otherwise reject them.
avro_null_as_default Bool 0 Replace input NULL in a non-nullable native target with its SQL DEFAULT or native type default; legacy SQL columns retain NULL.
avro_string_columns String '' Regular expression matching output String column paths to encode as UTF-8 string instead of bytes; empty matches none.
snappy_mode String basic Read Snappy client files as basic (Hadoop blocks) or framed (standard streaming format).

Export and import ​

Start a compatible schema registry first. Set the URL below to that registry; this example uses a writable subject named docs-example and a fresh destination table.

sh
export COORDINATOR_SCHEMA_REGISTRY_URL=http://127.0.0.1:8081
./coordinator client --database data --query "SELECT CAST(7 AS Int64) AS id SETTINGS avro_confluent_registry_url='$COORDINATOR_SCHEMA_REGISTRY_URL',avro_confluent_subject='docs-example' FORMAT avro_confluent" > sample.avro_confluent
./coordinator client --database data --query "CREATE TABLE registry_import(id Int64)" > /dev/null
./coordinator client --database data --query "INSERT INTO registry_import SETTINGS avro_confluent_registry_url='$COORDINATOR_SCHEMA_REGISTRY_URL' FORMAT avro_confluent" < sample.avro_confluent > /dev/null
./coordinator client --database data --query "SELECT id FROM registry_import FORMAT csv"
text
7