Enum EmbeddingsSource 

pub enum EmbeddingsSource {
    Hdf5(H5Dataset),
    Zarr {
        store: ReadableListableStorage,
        path: String,
    },
    Memory(Array2<f32>),
}

The embeddings a build reads, from an .h5 or .zarr file or an Array2<f32> already in memory. Files may store f32, f16, uint8 or int8, but read_vecs always returns f32 and only reads the rows asked for.

Variants

Hdf5(H5Dataset)

Zarr

Fields

store: ReadableListableStorage

path: String

Memory(Array2<f32>)

Already in memory; read_vecs just copies a slice.

Implementations

impl EmbeddingsSource

pub fn open(path: &Path, name: &str) -> Result<Self>

Opens the dataset (.h5) or array (.zarr) called name in the file at path, picking the format from path’s extension.

pub fn from_zarr(store: ReadableListableStorage, path: String) -> Self

Creates an embeddings source from the zarr array at path in store.

pub fn shape(&self) -> Result<(usize, usize)>

Returns the source’s size as (total_items, dim).

pub fn natural_chunk_vecs(&self, fallback: usize) -> Result<usize>

Returns how many vectors one on-disk chunk holds, or fallback if the source isn’t chunked.

pub fn chunk_aligned_batch_vecs( &self, memory_floor_vecs: usize, fallback: usize, ) -> Result<usize>

Picks a read batch size that covers whole chunks. Rounds memory_floor_vecs, the batch size the memory budget allows, up to whole chunks, so no chunk is split and decoded twice. fallback works as in natural_chunk_vecs.

pub fn native_dtype(&self) -> Result<EmbeddingDtype>

Returns the dtype the embeddings are stored as, always F32 for Memory.

pub fn read_vecs(&self, start: usize, end: usize) -> Result<Array2<f32>>

Reads vectors start..end as f32.