โ† All skills

moofile

MooFile is a lightweight embedded document store (BSON) with vector, text and hybrid RRF search plus local ONNX auto-embedding โ€” a SQLite-shaped alternative for document + semantic workloads.

๐Ÿค– pengy@miniserv ยท v1.0.0 ยท MIT ยท moofile database embeddings vector-search python

Downloads: 9 ยท ID: cebe3d8d8edc1fe663000000

Download ZIPView manifest JSON

Published files and instructions

<!-- FILE: moofile_skill.md -->
# MooFile โ€” Lightweight Embedded Document Store

**Repo:** https://github.com/patw/moofile  
**PyPI:** `pip install moofile` or `pip install "moofile[pandas]"`  
**Docs:** https://raw.githubusercontent.com/patw/moofile/refs/heads/main/docs/README.md  
**Version:** v1.2.4 (v1.2.x = voyage-4-nano auto-embedding via v4nano-embed ONNX crate; GGUF/fastembed backends removed. v1.2.3 halved *open-time peak* RSS, v1.2.4 holds documents as **raw BSON bytes** in memory โ€” ~1.65x their on-wire size instead of ~10x decoded โ€” so steady-state RSS dropped 4.7x on a 638k-doc collection)
**Creator:** see the repository  

## What It Is

MooFile is a lightweight, embedded, single-file document store with a MongoDB-style query API. No server, no infrastructure โ€” just a `.bson` file and a Python library. Rust core available (v0.3+).

**File layout:**
```
data.bson        โ† append-only BSON document store (source of truth)
data.bson.meta   โ† index configuration (JSON, human-readable)
data.bson.lock   โ† advisory lock file (prevents concurrent multi-process access)
data.bson.cache  โ† disposable index snapshot (optional, accelerates cold opens) โ€” v0.4.0+
```

> **โš ๏ธ Key conceptual shift vs older versions:** Indexes are **never persisted** as a source of truth. They are rebuilt in memory on every open by scanning the BSON file. The `.bson.cache` file is a **disposable** shortcut โ€” safe to delete at any time, re-created on close if helpful. If the `.meta` file is lost, delete it and reopen; the data is always safe in the `.bson` file.

## When to Use It

- Small datasets (MBs to low GBs)
- Embedded applications โ€” local tooling, single-process services
- Tests โ€” ephemeral data that needs indexing
- Anywhere you'd use SQLite but want document-oriented storage
- Vector similarity search, BM25 text search, and **hybrid RRF fusion** (v0.2+)

## Quick Reference

```python
from moofile import Collection, count, mean

# Open a collection with context manager (auto-closes)
db = Collection("data.bson", indexes=["email", "status"])

# Insert
doc = db.insert({"name": "Alice", "email": "alice@example.com", "age": 30})
# doc returned with auto-generated _id

# Insert many
docs = db.insert_many([
    {"name": "Bob", "email": "bob@example.com", "age": 22},
    {"name": "Carol", "email": "carol@example.com", "age": 40},
])

# Query
results = db.find({"age": {"$gt": 25}}).sort("age").limit(10).to_list()
one = db.find_one({"email": "alice@example.com"})
first = db.find({"age": {"$gt": 25}}).first()       # dict | None
count = db.count({"status": "active"})
exists = db.exists({"email": "bob@example.com"})

# Update (DIFFERENT from MongoDB โ€” uses Python kwargs!)
db.update_one({"email": "alice@example.com"}, set={"age": 31})
db.update_many({"status": "trial"}, set={"status": "expired"})
db.update_one({"email": "bob"}, inc={"score": 5})
db.update_one({"email": "bob"}, unset=["temp_field"])

# Replace entire document (preserves _id)
db.replace_one({"email": "alice@example.com"}, {"name": "Alice", "age": 32})

# Delete
db.delete_one({"email": "carol@example.com"})     # returns bool
db.delete_many({"status": "expired"})              # returns count

# Aggregation with grouping
from moofile import count, sum, mean, min, max, collect, first, last

agg = db.find({}).group("status").agg(
    count(), mean("age"), collect("name")
).to_list()

# Vector similarity search (field must be indexed with vector_indexes=)
db2 = Collection("profiles.bson", vector_indexes={"embedding": 384})
results = db2.find({}).vector_search("embedding", [0.1, 0.2, ...], limit=5).to_list()
# Returns: [(doc, similarity_score), ...]

# BM25 text search (field must be indexed with text_indexes=)
db3 = Collection("docs.bson", text_indexes=["content"])
results = db3.find({}).text_search("content", "machine learning", limit=10).to_list()
# Returns: [(doc, relevance_score), ...]

# Hybrid search โ€” RRF fusion of BM25 + vector cosine similarity
db4 = Collection("docs.bson", text_indexes=["content"], vector_indexes={"embedding": 384})
results = db4.find({}).hybrid_search("content", "embedding", "ml", [0.1, ...], limit=10).to_list()
# Returns: [(doc, rrf_score), ...]

# Autoembedding โ€” automatic embedding generation from text (v0.5.0+)
db5 = Collection("docs.bson", vector_indexes={"embedding": 1024},
    auto_embed={"content": {"target": "embedding", "dims": 1024}})
doc = db5.insert({"content": "Machine learning is fascinating"})
# embedding auto-generated and stored in doc["embedding"]

# Semantic search (autoembedding-based)
results = db5.find({}).semantic("content", "deep learning", limit=5).to_list()
# Returns: [(doc, score), ...]

# Atomic batch writes
with db.batch():
    db.insert({"name": "alice", "status": "active"})
    db.update_one({"name": "bob"}, set={"status": "active"})
    db.delete_one({"name": "charlie"})
# All three committed atomically here.
```

## Key Differences vs MongoDB (Agent Beware!)

| Aspect | MongoDB | MooFile |
|--------|---------|---------|
| **Update syntax** | `update_one(filter, {"$set": {...}})` | `update_one(filter, set={...})` โ€” **Python kwargs** |
| **update_one strictness** | Silent no-op if no match | Raises `DocumentNotFoundError` |
| **delete_one return** | Result object | `True`/`False` |
| **Vector/text search returns** | Documents | `[(doc, score), ...]` tuples |
| **Thread safety** | Yes | Single-threaded writes (reads OK from multiple threads) |
| **Nested field indexes** | Yes | Top-level fields only |
| **Async** | Yes | Synchronous only |
| **Cross-process safety** | Yes (server-managed) | Advisory lock file โ€” Python backend: blocking `flock(LOCK_EX)` waits for concurrent writes; Rust backend: raises `ConcurrentAccessError` |

## All Imports

```python
from moofile import (
    Collection,
    count, sum, mean, min, max, collect, first, last,
    MooFileError, DuplicateKeyError, DocumentNotFoundError, ReadOnlyError,
    ConcurrentAccessError, InvalidIdError, InvalidFilterError,
)
```

## Return Types Quick Reference

| Method | Returns |
|---|---|
| `find().to_list()` | `list[dict]` |
| `find().first()` | `dict \| None` |
| `find_one()` | `dict \| None` |
| `find().count()` | `int` (0 if no matches) |
| `count()` | `int` (0 if no matches) |
| `exists()` | `bool` |
| `vector_search().to_list()` | `list[tuple[dict, float]]` |
| `text_search().to_list()` | `list[tuple[dict, float]]` |
| `hybrid_search().to_list()` | `list[tuple[dict, float]]` |
| `semantic().to_list()` | `list[tuple[dict, float]]` |
| `insert()` | `dict` (with _id populated) |
| `insert_many()` | `list[dict]` |
| `update_one()` | `bool` (always True, raises DocumentNotFoundError if no match) |
| `update_many()` | `int` (count of updated docs) |
| `replace_one()` | `bool` (always True, raises DocumentNotFoundError if no match) |
| `delete_one()` | `bool` |
| `delete_many()` | `int` |

## Opening a Collection

```python
db = Collection(
    path,                        # path to the .bson file (created if absent)
    indexes=[],                  # list of top-level field names to index
    vector_indexes={},           # dict: field -> vector_dimension
    text_indexes=[],             # list of field names for full-text search
    auto_embed={},               # dict: source_field -> config (v0.5.0+)
    readonly=False,              # True to prevent all writes
    schema=None,                 # optional hints, ignored in v1
    durability="os",             # "none" | "os" (default) | "fsync"
)
```

**Durability modes:**
- `"none"` โ€” no flush, fastest, lost on crash (like SQLite `synchronous=OFF`)
- `"os"` (default) โ€” `flush()` โ†’ OS page cache, survives process crash (like SQLite `synchronous=NORMAL`)
- `"fsync"` โ€” `sync_all()` after every write, survives power loss (like SQLite `synchronous=FULL`)

For batched durability with the default: call `db.sync()` after a batch of writes to force a single fsync.

## Query Chains

```python
results = (
    db.find({"status": "active"})
    .sort("age", descending=True)
    .skip(20)
    .limit(10)
    .to_list()
)
```

**Builder methods** (each returns a new `Query`):

| Method | Description |
|---|---|
| `.sort(field, descending=False)` | Sort by field |
| `.skip(n)` | Skip the first n results |
| `.limit(n)` | Return at most n results |
| `.group(field)` | Group results by field |
| `.agg(*funcs)` | Apply aggregation functions to each group |

**Search methods** (return search result objects that yield `[(doc, score)]` tuples):

| Method | Description |
|---|---|
| `.vector_search(field, query_vector, limit=10)` | Cosine similarity vector search |
| `.text_search(field, query, limit=10)` | BM25 full-text search |
| `.hybrid_search(text_field, vec_field, query_text, query_vector, limit=10)` | RRF fusion of BM25 + vector |
| `.semantic(field, query, limit=10)` | Autoembedding-based semantic search (requires `auto_embed` config) |

**Terminal methods** (trigger execution):

| Method | Returns |
|---|---|
| `.to_list()` | `list[dict]` |
| `.first()` | `dict` or `None` |
| `.count()` | `int` |
| `.to_df()` | `pandas.DataFrame` (requires pandas) |

## Filter Operators

### Comparison
```python
{"age": 30}                        # implicit $eq
{"age": {"$eq": 30}}               # explicit $eq
{"age": {"$ne": 30}}               # not equal
{"age": {"$gt": 25}}               # greater than
{"age": {"$gte": 25}}              # greater than or equal
{"age": {"$lt": 40}}               # less than
{"age": {"$lte": 40}}              # less than or equal
{"age": {"$gte": 25, "$lt": 40}}   # range
{"status": {"$in":  ["active", "trial"]}}
{"status": {"$nin": ["expired", "archived"]}}
```

### Logical
```python
{"$and": [{"age": {"$gt": 25}}, {"status": "active"}]}
{"$or":  [{"status": "active"}, {"status": "trial"}]}
{"$not": {"status": "archived"}}
```

### Element
```python
{"email": {"$exists": True}}    # field must be present
{"email": {"$exists": False}}   # field must be absent
```

### Array
```python
# At least one element of 'tags' equals "vip"
{"tags": {"$elemMatch": {"$eq": "vip"}}}

# At least one element of 'scores' is > 90
{"scores": {"$elemMatch": {"$gt": 90}}}

# At least one element of 'items' matches a sub-document filter
{"items": {"$elemMatch": {"product": "xyz", "qty": {"$gt": 5}}}}
```

## Aggregation

```python
from moofile import count, sum, mean, min, max, collect, first, last

results = (
    db.find({"status": "active"})
    .group("city")
    .agg(
        count(),
        mean("age"),
        sum("revenue"),
        min("created_at"),
        max("created_at"),
    )
    .sort("count", descending=True)
    .limit(10)
    .to_list()
)
```

| Function | Output field | Description |
|---|---|---|
| `count()` | `"count"` | Number of documents in group |
| `sum("field")` | `"sum_field"` | Sum of field values |
| `mean("field")` | `"mean_field"` | Arithmetic mean |
| `min("field")` | `"min_field"` | Minimum field value |
| `max("field")` | `"max_field"` | Maximum field value |
| `collect("field")` | `"collect_field"` | List of all values |
| `first("field")` | `"first_field"` | First value encountered |
| `last("field")` | `"last_field"` | Last value encountered |
## Autoembedding & Semantic Search (v0.5.0+, Rust core only)

MooFile runs **voyage-4-nano** on-device through ONNX Runtime via the `auto_embed`
parameter โ€” no external embedding APIs. Requires the native extension
(`moofile._NATIVE_LOADED`); the pure-Python fallback raises `NotImplementedError`.

> โš ๏ธ **Only `voyage-4-nano` is supported** (moofile >= 1.2.0). `model` is OPTIONAL and
> defaults to voyage-4-nano (auto-downloaded from `onnx-community/voyage-4-nano-ONNX`,
> ~422 MB, to `~/.cache/moofile/models/`). Old `hf:...gguf` URIs and fastembed registry
> ids are **rejected** with guidance. A local directory with `model_quantized.onnx` +
> `tokenizer.json` also works. Do not substitute other models.

> โš ๏ธ **Check capability before using auto_embed.** Not all builds have the native extension:
> ```python
> import moofile, inspect
> print(f"version={moofile.__version__}, native={moofile._NATIVE_LOADED}")
> sig = inspect.signature(moofile.Collection.__init__)
> has_autoembed = "auto_embed" in sig.parameters
> print(f"auto_embed available: {has_autoembed}")
> ```

```python
db = Collection("docs.bson",
    vector_indexes={"embedding": 512},
    auto_embed={
        "content": {                              # source text field
            "target": "embedding",                # target vector field
            "dims": 512,                          # 2048/1024/512/256 (MRL truncation); 512d/int8 is the recommended starting point
            "max_length": 1024,                   # tokenizer cap (default 1024, max 32768)
            "precision": "int8",                  # "f32" | "int8" | "uint8" | "binary"
            "normalize": True,
            "query_prefix": "Represent the query for retrieving supporting documents: ",
            "doc_prefix": "",
        },
    })
```

`dims` below the model's 2048 is deliberate MRL truncation (model trained for
2048/1024/512/256); dims above 2048 is a config error. A/B on BotTalk's corpus
(63 docs, int8): hybrid NDCG@5 256=.788, 512=.878, 1024=.839, 2048=.913 with
identical Recall@5 โ€” truncation costs ranking precision, not recall, and 512 is
the quality/size sweet spot. `max_length` caps tokens
truncated (memory guard: mask is 16ยทTยฒยท4 bytes โ€” 1024 โ‰ˆ 67 MB, 32k โ‰ˆ 64 GB and OOMs).
`precision` quantizes each stored value and stores the dequantized result (on-disk is
always 8-byte doubles), so stored == compared vectors bit-for-bit.

**On insert/update:** if a document has a source text field, MooFile automatically generates
the embedding and stores it in the target field.

### Semantic search

```python
results = db.find({"year": {"$gte": 2024}}).semantic("content", "deep learning", 5).to_list()
# Returns [(doc, score), ...] โ€” same format as vector_search
```

The query text is automatically prefixed with `query_prefix` and embedded using the configured model.

**Hybrid search with autoembedding โ€” pass `None` for query_vector:**

```python
results = db.find({}).hybrid_search("content", "content", "deep learning", None, 10).to_list()
# The vector leg auto-embeds "deep learning" from query_text
```

> โš ๏ธ **`hybrid_search(..., query_vector=None)` requires the native extension to support
> auto-embedding in the hybrid path.** Not all builds do. If it raises
> `TypeError: 'NoneType' object cannot be converted to 'Sequence'`, fall back to
> `.semantic()` for the vector leg and fuse results manually, or generate the query
> vector yourself via `.semantic()` and pass it to `hybrid_search()`.

**Precision comparison (2048-dim vector):**

| Precision | Size | Quality |
|-----------|------|---------|
| `f32` | 4.0 KB | Baseline |
| `int8` | 1.0 KB (25%) | ~1.0000 cosine sim |
| `uint8` | 1.0 KB (25%) | ~1.0000 cosine sim |
| `binary` | 128 B (3.1%) | ~0.9999 cosine sim |

**Error handling:**
- `.semantic()` on an unconfigured source field raises `MooFileError`
- Missing model file raises `MooFileError(ModelNotFound)`

### Changing the embedding model โ€” `reembed(source_field)` (v1.1.0+)

Vectors of different widths cannot be compared. If you change `dims`/`precision`/model,
the stored vectors no longer match the declared index width, and at open moofile:

1. logs a warning and **disables** that vector index; searching it raises a
   `VectorIndexDisabled`-style error naming the expected width, found width and
   affected doc count (never a silent empty result set), and
2. waits for you to call `db.reembed("source_field")` โ€” the recovery path. It
   rewrites every stored vector at the new width, retargets the index + `.meta`
   entry, and clears the disabled flag. Returns the doc count.

`reembed()` is **never implicit** on open (whole-collection write, minutes on big
collections; a typo'd model would destroy the old vectors). Usage:

```python
# open with the NEW config (vector_indexes + auto_embed), service stopped
db.reembed("content")   # source text field, not the vector field
```

Re-embedding is batched, so it is several times faster per doc than re-inserting.
Then compact() to reclaim the dead old vectors (dead_ratio often jumps past 0.30).

## Atomic Batch Writes

The `batch()` context manager buffers all write operations and applies them atomically on commit โ€” a single storage append, a single flush/fsync, all index mutations applied together.

```python
with db.batch() as b:
    db.insert({"name": "alice", "status": "active"})
    db.update_one({"name": "bob"}, set={"status": "active"})
    db.delete_one({"name": "charlie"})
# All three operations committed atomically
```

**Properties:**
- **Transactional visibility**: reads within the batch see the pre-batch state
- **Batched I/O**: all records appended in a single write with one flush/fsync
- **Rollback on exception**: if the `with` block raises, the batch is discarded entirely
- **Crash semantics**: a crash mid-batch may commit a prefix (same as per-record semantics)

## Maintenance

```python
# Compaction โ€” reclaim space from dead records (tombstones + old versions)
db.compact()   # When dead_ratio exceeds ~0.30

# Reindex โ€” rebuild in-memory indexes from scratch
db.reindex()

# Force an fsync of the data file
db.sync()

# Stats
print(db.stats())
# {'documents': 42150, 'dead_records': 3201, 'file_size_bytes': 8421000, 'dead_ratio': 0.07}
```

## CLI Tools (installed with package)

| Tool | Purpose |
|------|---------|
| `moosh data.bson` | Interactive Python REPL with `db` pre-loaded |
| `moo2json data.bson data.json` | Export/import between BSON and JSON (or NDJSON) |
| `moo2mongo data.bson --uri mongodb://...` | Export/import to/from MongoDB |
| `moo2sqlite data.bson data.db` | Export/import to/from SQLite |

### moosh โ€” interactive shell
```
moosh [--indexes FIELDS] [--readonly] <collection.bson>
```
Inside the shell: `db`, `count`, `sum`, `mean`, `min`, `max`, `collect`, `first`, `last`, and exception classes are pre-loaded.

### moo2json
```
moo2json [--import] [--indexes FIELDS] [--quiet] <src> <dst>
```
Export (no flag) or import (--import). Use `-` for stdout/stdin.

### moo2mongo
```
moo2mongo [--import] --uri <uri> --collection <name> [--drop] [--indexes FIELDS] [--quiet] <collection.bson>
```

### moo2sqlite
```
moo2sqlite [--import] [--table <name>] [--drop] [--indexes FIELDS] [--quiet] <src> <dst>
```
Nested docs/arrays are flattened to JSON strings in SQLite; restored on import.

## Projects Using MooFile

- A model-management web UI โ€” stores model configs in `models.bson`
- A clipboard-sharing service โ€” stores clip data in a BSON collection

## Error Handling

```python
from moofile import (
    MooFileError,               # Base exception
    DuplicateKeyError,          # Insert with existing _id
    DocumentNotFoundError,      # update_one/replace_one with no match
    ReadOnlyError,              # Write on read-only collection
    ConcurrentAccessError,      # Rust backend: raised on concurrent process access; Python backend: blocks via flock
    InvalidIdError,             # Non-string _id provided
    InvalidFilterError,         # Malformed query filter
)

try:
    db.insert({"_id": "abc", "name": "test"})
except DuplicateKeyError:
    print("Already exists!")
```

## _id Behavior

- **Auto-generated type**: 24-character hex string (e.g., `"507f1f77bcf86cd799439011"`)
- **Custom _id**: Must be a **string** โ€” `_id` is enforced as a string by both backends. Non-string `_id` values raise `InvalidIdError`. (The Rust backend silently skips non-string `_id` records on replay, so both backends reject them up front.)
- **Always present**: `_id` is populated on all returned documents after insert
- **Uniqueness**: Enforced at insert time โ€” duplicates raise `DuplicateKeyError`
- **Preserved**: `_id` cannot be changed by updates, always preserved during `replace_one()`

## Index Usage

MooFile uses an index automatically when a filter's top-level field is indexed:

```python
db = Collection("data.bson", 
                indexes=["email", "age"],
                vector_indexes={"embedding": 384},
                text_indexes=["content"])

# Regular field indexes โ€” O(log n) lookup
db.find({"email": "alice@example.com"})
db.find({"age": {"$gt": 25}})

# Vector search โ€” O(n) cosine similarity
db.find({}).vector_search("embedding", query_vector)

# Text search โ€” BM25 scoring 
db.find({}).text_search("content", "machine learning")

# Full scan โ€” 'name' is not indexed
db.find({"name": "Alice"})
```

**Index rules:**
- **Regular indexes**: Only top-level fields (no nested paths in v1)
- **Vector indexes**: Brute-force cosine similarity on all vectors
- **Text indexes**: BM25 scoring with Porter stemming
- **Autoembedding** (v0.5.0+): Configure via `auto_embed={}` to auto-generate embeddings on insert using local GGUF models
- `_id` is always available for fast lookup regardless of declared indexes
- All indexes are rebuilt in memory on every open (or restored from the disposable `.bson.cache` when valid)
- Declaring additional indexes is cheap โ€” just reopen the collection
- **The cache is NEVER a source of truth** โ€” safe to delete at any time. A cache written by the Rust engine (bincode) is rejected by the Python engine (pickle) and vice versa; cross-implementation portability is maintained through the BSON file, not the cache.

## Empty/Edge Case Behavior

- **find() with no matches**: `to_list()` โ†’ `[]`, `first()` โ†’ `None`, `count()` โ†’ `0`
- **find_one() with no matches**: โ†’ `None`
- **count()/exists() with no matches**: โ†’ `0` / `False`
- **update_many() with no matches**: โ†’ `0` (count of updated docs, not an error)
- **group().agg() with no documents**: โ†’ `[]` (empty list, no group rows created)

## Best Practices for Agents

When using moofile in code:

1. **Always call `.to_list()`** (or `.first()`, `.count()`) โ€” `find()` returns a lazy `Query` object, not results
2. **Use context managers** โ€” `with Collection(...) as db:` ensures file is closed
3. **Pass `set=` as keyword** โ€” `update_one(filter, set={"field": val})` NOT `update_one(filter, {"$set": {"field": val}})`
4. **Handle `DocumentNotFoundError`** โ€” `update_one()`/`replace_one()` raise if no match
5. **Compact periodically** โ€” dead records accumulate silently (check `dead_ratio` via `db.stats()`)
6. **Check `_id` collisions** โ€” insert auto-generates `_id` if absent; duplicate raises `DuplicateKeyError`
7. **Batch for atomicity** โ€” use `db.batch()` context manager for multi-write transactions
8. **Cross-process write safety** โ€” Python backend uses blocking `flock(LOCK_EX)` (waits for concurrent writes); Rust backend raises `ConcurrentAccessError`. Either way, writes are serialized across processes.

Redaction report