Index Format#

An index is a directory holding a Zarr store. Every part of it is a zarr array, so any Zarr reader can inspect it.

See Zarr Concepts first for what a Zarr array, group, and chunk actually are. This page covers what ecpfs specifically stores in them.

Version 1#

This page describes format version 1, stored in info/format_version. The 0.10.x releases keep this format and the public API unchanged.

Every version 1 index has these arrays:

  • info/format_version, info/levels, info/metric, info/total_items and info/next_item_id

  • rep_item_ids

  • index_root/embeddings

  • embeddings and either node_ids or item_ids in each node that exists

Every other array is optional. Loaders read an index without it, and a later release can add optional arrays without changing the version. The optional arrays today are info/is_normalized, rep_embeddings, each node’s border and everything under queries/.

Settings#

info/ holds one scalar per setting.

  • info/format_version (uint32): the version of this layout, currently 1. Index refuses any other value. If the field is missing from an index that is otherwise complete, Index adds it when it opens the index.

  • info/levels (uint32): node levels below the root. The last one holds the leaf nodes.

  • info/metric (string): "L2" or "IP".

  • info/is_normalized (bool, optional): whether every embedding is unit-length. An index without it is searched as not normalized.

  • info/total_items (uint32): items stored.

  • info/next_item_id (uint32): the id the next insert gives out. Equal to total_items unless a crash stopped an insert partway.

Representatives#

  • rep_embeddings, shape (num_representatives, dim), optional: every representative, the items that every other item is clustered around.

  • rep_item_ids, shape (num_representatives,), uint32: each representative’s item id.

  • index_root/embeddings, shape (node_size, dim): the root node, which holds the first node_size representatives.

Tree nodes#

Level N of the tree lives under lvl_N/, with one group per node.

  • lvl_N/node_M/embeddings, shape (n, dim): the representatives of node M’s children, or on the leaf level, the items’ own embeddings.

  • lvl_N/node_M/node_ids, shape (n,), uint32: the child node ids on level N+1. Internal levels only.

  • lvl_N/node_M/item_ids, shape (n,), uint32: the item ids. Leaf level only.

  • lvl_N/node_M/border, shape (2,), float32, optional: created, never populated.

n differs from node to node, since eCP does not enforce cluster sizes. A node that received no children during the build is never written, so node ids on a level can have gaps.

Embeddings arrays are stored as float32, float16, uint8 or int8, chosen at build time with EmbeddingDtype. Every read widens them to float32.

Chunking#

Every array carries its own chunk shape, fixed when the array is created. The representative arrays take rep_chunk_bytes and the tree nodes take node_chunk_bytes, both given to Builder at build time and counted in the stored dtype.

A node holding fewer rows than its chunk still declares the full shape. That costs nothing on disk, since the unused part compresses away, but reading the node decodes the whole chunk. A node holding more rows spans several chunks.

An index keeps the chunk shape it was built with, so changing either setting affects new indexes only.

Saved queries#

queries/ is optional. queries/Q/ holds query Q’s saved state, written by close() or when the query is evicted from memory.

  • query, shape (dim,), float32: the query vector.

  • tree_pq_score (float32), tree_pq_is_leaf (int32), tree_pq_level and tree_pq_node_id (uint32), each shape (n,): the queue of nodes still to visit, split into one array per field.

  • items_score (float32) and items_id (uint32), each shape (m,): the items found but not yet returned.

  • exclude, shape (e,), uint32, optional: the query’s exclude set. Written only when the set is not empty.

  • persisted_at (uint64): the save time in Unix seconds.