Skip to content
Coordinator DB
View as Markdown
View as Markdown

arrow_stream ​

Description ​

Arrow IPC stream with a schema, record batches and an end marker.

Input and output. Select it with FORMAT arrow_stream. Names ignore ASCII case; use the snake_case name shown here.

Use this layout with consumers expecting sequential Arrow IPC messages rather than a seekable IPC file. A stream begins with its schema, can contain several record batches and must end cleanly; trailing bytes after its end marker are rejected.

The value mappings are the same as arrow. Use explicit target declarations when importing and the native client exact-byte mode when exporting.

text
SELECT expressions FORMAT arrow_stream;
INSERT INTO table [(columns)] [SETTINGS ...] FORMAT arrow_stream
<data>

Example usage ​

Use the native client's --query mode to preserve binary output bytes. The server must be running, the executable available as ./coordinator, and database data selected.

sh
./coordinator client --database data --query "SELECT CAST(7 AS Int64) AS id FORMAT arrow_stream" > sample.arrow_stream
./coordinator client --database data --query "CREATE TABLE format_import(id Int64)" > /dev/null
./coordinator client --database data --query "INSERT INTO format_import FORMAT arrow_stream" < sample.arrow_stream > /dev/null
./coordinator client --database data --query "SELECT id FROM format_import FORMAT csv"
text
7

Create format_import in a fresh database before running this example.

Format settings ​

Use a statement SETTINGS clause; these options are not persisted by SET.

Setting Type Default Meaning
arrow_compression String lz4_frame Compress record batch buffers with lz4_frame or zstd; none writes uncompressed buffers.
arrow_date_as_number Bool 0 Write Date as UInt16 days since the epoch instead of Date32; Date32 always remains Date32.
arrow_dictionary Bool 0 Write LowCardinality as dictionary indices and values instead of expanding its scalars.
arrow_dictionary_64bit_indices Bool 0 Use 64-bit dictionary indices instead of the smallest width that fits the dictionary.
arrow_dictionary_signed_indices Bool 1 Use signed dictionary indices; when off, use unsigned indices.
arrow_fixed_strings Bool 1 Write FixedString with a fixed byte width instead of variable-length Binary.
arrow_ignore_case Bool 0 Ignore ASCII case when matching input field names to target columns.
arrow_missing_as_default Bool 1 Give target columns absent from the input schema their DEFAULT; when off, reject missing columns.
arrow_strings_as_text Bool 1 Write text strings as Utf8 instead of Binary; byte strings keep their exact bytes.
ignore_unknown_fields Bool 0 Skip input keys and header names that match no target column instead of failing.
snappy_mode String basic Read Snappy client files as basic (Hadoop blocks) or framed (standard streaming format).