semantic-chunking
NPM Package for Semantically creating chunks from large texts. Useful for workflows involving large language models (LLMs).
An NPM package for semantically creating chunks from large texts, aimed at workflows involving large language models. It splits the input into sentences, generates a vector for each using a specified ONNX model, calculates cosine similarity for each sentence pair, and groups sentences into chunks according to a similarity threshold and a maximum token size. Adjacent chunks that are similar can optionally be rebalanced and combined into larger ones up to that maximum, and the final chunks are returned as an array of objects. Configuration covers dynamic similarity thresholds, chunk sizes, multiple embedding model options, quantized model support, and chunk prefixes for RAG workflows. A web UI for experimenting with settings is included and can be run through Docker Compose, alongside a hosted online demo.
Installation
npm install semantic-chunkingRelated\\
chunk-match
A NodeJS library that semantically chunks text and matches it against a user query using cosine similarity for precise and relevant text retrieval.
embedding-utils
Vector math, similarity search, ANN indexing, clustering, async pipelines, evaluation metrics, and multi-provider embedding generation -- zero dependencies,…
extract-topics
Use LDA (Latent Dirichlet Allocation) to extract topics from text