llm-distillery
Use LLMs to distill large texts down to a manageable size by utilizing a map-reduce approach. This ensures that the text fits within a specified token limit,...
It reduces text size based on token count so the result fits within a specified token limit for downstream LLM tasks. It uses the semantic-chunking library to split text into chunks that are then summarized, with customizable parameters like target token size and chunking thresholds.
Related \\
-
chunk-match
A NodeJS library that semantically chunks text and matches it against a user query using cosine similarity for precise and relevant text retrieval.
-
embedding-utils
Vector math, similarity search, ANN indexing, clustering, async pipelines, evaluation metrics, and multi-provider embedding generation -- zero dependencies,...
-
extract-topics
Use LDA (Latent Dirichlet Allocation) to extract topics from text