# avro_confluent

## Description

Consecutive Avro records prefixed with a zero byte and a big-endian registry schema ID. Configure avro_confluent_registry_url; output also needs avro_confluent_subject. This is a schema-registry format, rather than an Avro object container.

Input and output. Select it with `FORMAT avro_confluent`. Names ignore ASCII case; use the snake_case name shown here.

Every record carries a schema identifier rather than an embedded object-container schema. Input resolves that identifier through the configured registry. Output registers or obtains a schema under the configured subject.

Supply a reachable registry URL on the statement. Output also needs a writable subject. These are external-service requirements; ordinary avro files use a different envelope.

```text
SELECT expressions FORMAT avro_confluent;
INSERT INTO table [(columns)] [SETTINGS ...] FORMAT avro_confluent
<data>
```

## Example usage

Use the native client's `--query` mode to preserve binary output bytes. The server must be running, the executable available as `./coordinator`, and database `data` selected.

Set `avro_confluent_registry_url` to your running schema registry and `avro_confluent_subject` to a writable subject before export. A registry-dependent command requires that external service.

```sql
SELECT name,is_input,is_output FROM system.formats WHERE name='avro_confluent' FORMAT csv;
```

```text
"avro_confluent",1,1
```

## Format settings

Use a statement `SETTINGS` clause; these options are not persisted by `SET`.

| Setting                          | Type     | Default | Meaning                                                                                                                         |
| -------------------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `avro_confluent_connect_timeout` | `UInt64` | `1`     | Connection and TLS handshake timeout in seconds; positive values are capped at 599.                                             |
| `avro_confluent_receive_timeout` | `UInt64` | `1`     | Registry response receive timeout in seconds; positive values are capped at 599.                                                |
| `avro_confluent_registry_url`    | `String` | `''`    | HTTP or HTTPS schema registry base URL; URL-encoded credentials select Basic authentication.                                    |
| `avro_confluent_retries`         | `UInt64` | `5`     | Retry transient registry failures this many times; zero disables retries and values above 20 are capped at 20.                  |
| `avro_confluent_retry_delay_ms`  | `UInt64` | `100`   | Initial retry delay in milliseconds, capped at 60000; subsequent delays double up to 10000 milliseconds.                        |
| `avro_confluent_send_timeout`    | `UInt64` | `1`     | Registry request send timeout in seconds; positive values are capped at 599.                                                    |
| `avro_confluent_subject`         | `String` | `''`    | Subject under which an output record schema is registered; required for avro_confluent output.                                  |
| `avro_missing_as_default`        | `Bool`   | `0`     | Give target fields absent from the input schema their SQL DEFAULT or native type default; otherwise reject them.                |
| `avro_null_as_default`           | `Bool`   | `0`     | Replace input NULL in a non-nullable native target with its SQL DEFAULT or native type default; legacy SQL columns retain NULL. |
| `avro_string_columns`            | `String` | `''`    | Regular expression matching output String column paths to encode as UTF-8 string instead of bytes; empty matches none.          |
| `snappy_mode`                    | `String` | `basic` | Read Snappy client files as basic (Hadoop blocks) or framed (standard streaming format).                                        |

## Export and import

Start a compatible schema registry first. Set the URL below to that registry; this example uses a writable subject named docs-example and a fresh destination table.

```sh
export COORDINATOR_SCHEMA_REGISTRY_URL=http://127.0.0.1:8081
./coordinator client --database data --query "SELECT CAST(7 AS Int64) AS id SETTINGS avro_confluent_registry_url='$COORDINATOR_SCHEMA_REGISTRY_URL',avro_confluent_subject='docs-example' FORMAT avro_confluent" > sample.avro_confluent
./coordinator client --database data --query "CREATE TABLE registry_import(id Int64)" > /dev/null
./coordinator client --database data --query "INSERT INTO registry_import SETTINGS avro_confluent_registry_url='$COORDINATOR_SCHEMA_REGISTRY_URL' FORMAT avro_confluent" < sample.avro_confluent > /dev/null
./coordinator client --database data --query "SELECT id FROM registry_import FORMAT csv"
```

```text
7
```