semantic-chunking
NPM Package for Semantically creating chunks from large texts. Useful for workflows involving large language models (LLMs).
An NPM package for semantically creating chunks from large texts, useful for workflows involving large language models (LLMs). It splits input text into sentences, generates a vector for each using a specified ONNX model, calculates cosine similarity for each sentence pair, and groups sentences into chunks based on a similarity threshold and maximum token size. Options include dynamic similarity thresholds, configurable chunk sizes, multiple embedding model options, quantized model support, and chunk prefixes for RAG workflows. It also includes a Web UI for experimenting with settings and can be run via Docker Compose.
Installation
npm install semantic-chunkingRelated \\
-
chunk-match
A NodeJS library that semantically chunks text and matches it against a user query using cosine similarity for precise and relevant text retrieval.
-
embedding-utils
Vector math, similarity search, ANN indexing, clustering, async pipelines, evaluation metrics, and multi-provider embedding generation -- zero dependencies,…
-
extract-topics
Use LDA (Latent Dirichlet Allocation) to extract topics from text