API Reference#

Note

These are documented under their compiled-extension names (ecpfs.ecp.*), which autodoc needs to find their members. Use them as ecpfs.Index and ecpfs.Builder, the names the package re-exports.

Index#

class ecpfs.ecp.IndexWrapper(index_path, memory_limit_bytes=Ellipsis)#

An eCP index opened from disk, for searching and inserting.

memory_limit_bytes caps the index data kept in memory, nodes and open queries together, and defaults to 80% of system RAM. Every method except set_memory_limit_bytes releases the GIL while it works, so other Python threads keep running. Use the index as a context manager, or call close, so unfinished queries are saved for later.

cleanup_persisted_queries_older_than(cutoff_unix_secs)#: Erases every query saved to disk before cutoff_unix_secs, a Unix timestamp such as datetime(...).timestamp(). Returns how many were erased.

close()#: Saves every open query to disk so a later Index can resume it, then closes this one, raising OSError if a save failed. Any other method raises ValueError afterwards. Safe to call more than once.

get_next_k_items(query_id, k, search_exp, max_increments, exclude_vec)#: Returns the next k (score, item_id) pairs of a query started with new_search, searching further if needed. Empty for an unknown query_id, and raises ValueError after close. See Search Parameters for the other arguments.

insert(embeddings)#: Adds each row of embeddings to its nearest leaf and returns the new ids as (start_id, end_id), so range(start_id, end_id) lists them in row order. Rows must match the index’s dimension, and be unit-length if it was built with is_normalized. Leaves only grow; the tree is never rebalanced.

new_search(query, k, search_exp, max_increments, exclude_vec)#: Searches for query and returns (items, query_id), where items are the k best (score, item_id) pairs, lowest score first. Pass query_id to get_next_k_items for more. See Search Parameters for the arguments and what a score means.

set_memory_limit_bytes(memory_limit_bytes)#: Changes the memory limit without reopening the index. Lowering it below what is cached evicts right away. Meant for occasional use, not per query.

Builder#

class ecpfs.ecp.BuilderWrapper(index_path, levels, metric, is_normalized=False, memory_limit_bytes=Ellipsis, embedding_dtype=None, rep_chunk_bytes=Ellipsis, node_chunk_bytes=Ellipsis)#

Builds a new eCP index in three steps: create the Builder, call select_representatives or select_representatives_custom, then call build. Every method releases the GIL while it works.

The index is created at index_path. levels is the number of node levels below the root, and is_normalized should be set only if every embedding is unit-length. memory_limit_bytes is a target for the build’s memory use rather than a hard cap, and defaults to 80% of system RAM. Leave embedding_dtype as None to store each file’s own dtype; a narrower one logs a warning. rep_chunk_bytes is the chunk size for the representative arrays and node_chunk_bytes the chunk size for the tree nodes. Measure zarr read speed at a few chunk sizes on your own data before changing them.

build(embeddings_file, fallback_batch_rows, grp_name='embeddings')#: Builds the tree from the dataset grp_name in embeddings_file, whose rows get item ids 0, 1, 2, … in order. fallback_batch_rows works as in select_representatives.

select_representatives(embeddings_file, target_cluster_items, strategy, fallback_batch_rows, grp_name='embeddings')#: Picks the representatives the tree is built from, out of the dataset grp_name in embeddings_file (.h5 or .zarr). strategy is “offset” (evenly spaced items) or “random”. target_cluster_items is the average cluster size to aim for, and fallback_batch_rows the chunk size assumed if the file isn’t chunked.

select_representatives_custom(ids, embeddings)#: Uses your own representatives instead of picking them, such as ones from an external clustering step. ids[i] is the id of row i of embeddings.

Metric#

class ecpfs.ecp.Metric#

How the distance between two vectors is measured: L2 (Euclidean distance, lower is closer) or IP (inner product, higher is closer).

IP = Metric.IP#

L2 = Metric.L2#

EmbeddingDtype#

class ecpfs.ecp.EmbeddingDtype#

The type an index stores its embeddings as on disk: F32, F16, UInt8 (0 to 255) or Int8 (-128 to 127). Every read widens to f32, so a narrower type saves disk, not memory.

F16 = EmbeddingDtype.F16#

F32 = EmbeddingDtype.F32#

Int8 = EmbeddingDtype.Int8#

UInt8 = EmbeddingDtype.UInt8#

Logging#

See Logging for the level policy, the JSONL record format, and where the file goes.

ecpfs.ecp.init_logging(log_dir=None, level='info')#: Starts logging this process to a new JSONL file in log_dir (default ecp_logs/) and returns the file’s path. level is “off”, “trace”, “debug”, “info”, “warn” or “error”; “off” creates the file but logs nothing. Only the first call sets logging up; later calls return the same path.