[!CAUTION] text-generation-inference is now in maintenance mode. Going forward, we will accept pull requests for minor bug fixes, documentation improvements and lightweight maintenance tasks.

TGI has initiated the movement for optimized inference engines to rely on a transformers model architectures. This approach is now adopted by downstream inference engines, which we contribute to and recommend using going forward: vllm, SGLang, as well as local engines with inter-compatibility such as llama.cpp or MLX.

<div align="center"> <a href="https://www.youtube.com/watch?v=jlMAX2Oaht0"> <img width=560 alt="Making TGI deployment optimal" src="https://huggingface.co/datasets/Narsil/tgi_assets/resolve/main/thumbnail.png"> </a>

Text Generation Inference

<a href="https://github.com/huggingface/text-generation-inference"> <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/huggingface/text-generation-inference?style=social"> </a> <a href="https://huggingface.github.io/text-generation-inference"> <img alt="Swagger API documentation" src="https://img.shields.io/badge/API-Swagger-informational"> </a>

A Rust, Python and gRPC server for text generation inference. Used in production at Hugging Face to power Hugging Chat, the Inference API and Inference Endpoints.

</div>

Table of contents

• Get Started
  • Docker
  • API documentation
  • Using a private or gated model
  • A note on Shared Memory (shm)
  • Distributed Tracing
  • Architecture
  • Local install
  • Local install (Nix)
• Optimized architectures
• Run locally
  • Run
  • Quantization
• Develop
• Testing

Text Generation Inference (TGI) is a toolkit for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation for the most popular open-source LLMs, including Llama, Falcon, StarCoder, BLOOM, GPT-NeoX, and more. TGI implements many features, such as:

• Simple launcher to serve most popular LLMs
• Production ready (distributed tracing with Open Telemetry, Prometheus metrics)
• Tensor Parallelism for faster inference on multiple GPUs
• Token streaming using Server-Sent Events (SSE)
• Continuous batching of incoming requests for increased total throughput
• Messages API compatible with Open AI Chat Completion API
• Optimized transformers code for inference using Flash Attention and Paged Attention on the most popular architectures
• Quantization with :
  • bitsandbytes
  • GPT-Q
  • EETQ
  • AWQ
  • Marlin
  • fp8
• Safetensors weight loading
• Watermarking with A Watermark for Large Language Models
• Logits warper (temperature scaling, top-p, top-k, repetition penalty, more details see transformers.LogitsProcessor)
• Stop sequences
• Log probabilities
• Speculation ~2x latency
• Guidance/JSON. Specify output format to speed up inference and make sure the output is valid according to some specs..
• Custom Prompt Generation: Easily generate text by providing custom prompts to guide the model's output
• Fine-tuning Support: Utilize fine-tuned models for specific tasks to achieve higher accuracy and performance

Hardware support

• Nvidia
• AMD (-rocm)
• Inferentia
• Intel GPU
• Gaudi
• Google TPU

Get Start