llm-distillery
Use LLMs to distill large texts down to a manageable size by utilizing a map-reduce approach. This ensures that the text fits within a specified token limit,...
Intelligently splits text via the semantic-chunking library, summarizes each chunk with an OpenAI-compatible LLM, then reduces until the output fits within a configured token budget. llmDistillery(text, options) takes parameters such as targetTokenSize, baseUrl, apiKey, llmModel, and stopTokens. Callable against any OpenAI-compatible endpoint (e.g., together.ai).
Related \\
-
chunk-match
A NodeJS library that semantically chunks text and matches it against a user query using cosine similarity for precise and relevant text retrieval.
-
embedding-utils
Vector math, similarity search, ANN indexing, clustering, async pipelines, evaluation metrics, and multi-provider embedding generation -- zero dependencies,...
-
extract-topics
Use LDA (Latent Dirichlet Allocation) to extract topics from text