Documentation

Factly is a modern CLI tool designed to evaluate the factuality of Large Language Models (LLMs) on the Massive Multitask Language Understanding (MMLU) benchmark. It provides a robust framework for prompt engineering experiments and factual accuracy assessment.

Overview

Features

• Evaluate LLM factuality on the MMLU benchmark with detailed results

• Support for various prompt engineering experiments via configurable system instructions

• Generate comparative visualizations of factuality scores across models and prompts

• Structured output for easy analysis and comparison

• Built with modern Python tooling (Python 3.12, uv, click, pydantic)

• Extensible and reproducible evaluation workflows

Note

Currently, only OpenAI models are supported.

Quick Start

# Run MMLU evaluation with default settings
factly mmlu

# Run MMLU evaluation and generate plots
factly mmlu --plot

# Get help on all available options
factly mmlu --help

# Get help on all available commands
factly --help

That’s it! The tool uses optimized default parameters and saves all outputs to the output directory.

Note

For detailed installation instructions, please see the Installation Guide. And for usage instructions, use cases, examples, and advanced configuration options, please see the Usage Guide.

---

Full Table of Contents

The User Guide

This part of the documentation, which is mostly prose, begins with some background information about Factly, then focuses on step-by-step instructions for getting the most out of Factly.

• Installation
  • Overview
  • Requirements
  • Installation Methods
  • Dependencies
  • Troubleshooting
• Usage
  • Command Line Interface
  • Advanced Usage
  • Examples
  • Environment Variables

The Community Guide

This part of the documentation, which is mostly prose, details the Factly ecosystem and community.

• Changelog
  • 1.1.0 - 2025-05-??
  • 1.0.1 - 2025-04-30
  • 1.0.0 - 2025-04-30

The API Documentation / Guide

If you are looking for information on a specific function, class, method, or algorithm, this part of the documentation is for you.

• API Reference
  • The factly module
    • The __main__ module
  • The factly.llms module
    • FactlyGptModel
    • FactlyModelMixin
    • The factly.llms.base_model module
    • The factly.llms.openai_model module
    • The cli module
    • The benchmarks module
    • The plots module
    • The resources module
    • The settings module
    • The tasks module

The Contributor Guide

If you want to contribute to the project, this part of the documentation is for you.

• Maintainers’ Guide
  • Overview
  • Development Environment
  • Testing the Project
  • Building the Package
  • Documentation Management
  • Linting and Code Quality Checks
  • Release Process
  • Continuous Integration and Deployment
  • Development Guidelines
  • Troubleshooting
• Contributing to Factly
  • Code of Conduct
  • Getting Started
  • Development Environment
  • Coding Standards
  • Testing
  • Pull Request Process
  • Project Structure
  • Questions?
  • References
• License and Credits
  • Credits
  • Acknowledgements

Support

Should you have any question, any remark, or if you find a bug, or if there is something you can’t do with the Factly, please open an issue.

Project Information

Factly is released under the MIT License, its documentation lives at Read the Docs, the code on GitHub, and the latest release on PyPI. It’s rigorously tested on Python 3.12+.

If you’d like to contribute to Factly you’re most welcome!

Indices and tables

• Index

• Module Index

• Search Page