semantic-chunking
NPM Package for Semantically creating chunks from large texts. Useful for workflows involving large language models (LLMs).
Splits input into sentences, generates embeddings using a configurable ONNX model, computes cosine similarity between pairs, and groups sentences into chunks by similarity threshold and max token size. Supports dynamic thresholds, chunk rebalancing, quantized models, and RAG-style chunk prefixes. Includes a Web UI for experimenting with settings, runnable via Docker Compose.
Related \\
-
chunk-match
A NodeJS library that semantically chunks text and matches it against a user query using cosine similarity for precise and relevant text retrieval.
-
embedding-utils
Vector math, similarity search, ANN indexing, clustering, async pipelines, evaluation metrics, and multi-provider embedding generation -- zero dependencies,...
-
extract-topics
Use LDA (Latent Dirichlet Allocation) to extract topics from text