k-batch
K-Batch is an intelligent text batching library that uses k-means clustering to group sentences by length for optimal processing. It's particularly useful for NLP tasks, machine learning batch processing, and any scenario where processing similar-length texts together improves efficiency.
A text batching library that uses k-means clustering to group sentences by length for optimal processing, aimed at NLP tasks and machine-learning batch processing where grouping similar-length texts improves efficiency. Grouping by length reduces padding waste, improves computational efficiency, optimizes memory usage, and ensures each batch meets minimum size requirements. The main kBatchSentences(sentences, options) function returns an array of batches, with options for maximum batches, minimum sentences per batch, the minimum sentences required to split, and maximum k-means iterations. A companion analyzeKBatches function returns statistics about the resulting batches.
Installation
npm install k-batchRelated \\
-
chunk-match
A NodeJS library that semantically chunks text and matches it against a user query using cosine similarity for precise and relevant text retrieval.
-
embedding-utils
Vector math, similarity search, ANN indexing, clustering, async pipelines, evaluation metrics, and multi-provider embedding generation -- zero dependencies,…
-
extract-topics
Use LDA (Latent Dirichlet Allocation) to extract topics from text