InferIQ

LLM Evaluation framework that leverages LLMs to Evaluate other LLMs

A Node.js LLM evaluation framework that uses LLMs to evaluate other LLMs. It generates answers to questions from a sample dataset across the LLMs in an evaluation pool, then has a group of Judge LLMs assess and rate each response. Results are visualized across several graphs to show overall accuracy, and additional metrics such as BERT Score and inference time are reported per LLM. It is set up with an npm setup script and started with npm run server or npm start.

Setup

  • run the npm setup script:
    npm run setup