---
title: "Tech Tip #32: Quantise before you buy more GPUs"
description: "8-bit weights ran 175-billion-parameter models without losing quality."
url: https://greencode.ai/tech-tips/32-quantise-before-you-scale/
---

# Tech Tip #32: Quantise before you buy more GPUs

**8-bit weights ran 175-billion-parameter models without losing quality.**

[Watch the video](https://media.greencode.ai/shorts/quantise-before-you-scale-a6754986ca08.mp4)

Video created by [explainapaper.com](https://explainapaper.com/?utm_source=greencode.ai&utm_medium=film&utm_campaign=made-by)

### The problem

Models are deployed in full precision by default, needing more memory, more GPUs and more energy per answer than the task requires.

### The fix you can use today

- Test 8-bit or 4-bit quantised versions against your quality bar.
- Use the quantised model where results hold up.
- Measure energy and latency per request before and after.

Tim Dettmers and colleagues showed that very large models can run on 8-bit numbers without losing quality. GreenCode's writing on green AI draws on this work.

### Tools that help

- [llama.cpp](https://github.com/ggml-org/llama.cpp?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): runs quantised open-weight language models on ordinary CPUs and GPUs
- [nvidia-smi](https://developer.nvidia.com/system-management-interface?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): reports the power draw of NVIDIA GPUs from the card's own sensors
- [AI Energy Score](https://huggingface.github.io/AIEnergyScore/?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode) (Hugging Face): comparable energy ratings for AI models on common tasks

### How GreenCode helps

GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, retrieval so a model reads only what it needs, and each request routed to the smallest component that can handle it. [More on this](https://greencode.ai/feature/green-optimised-generative-ai/).

### Tags

#llm#model-efficiency#gpu#inference

### Sources

- [Green AI is a mathematics problem (LLM.int8(), mixed precision, QLoRA)](https://greencode.ai/news/mathematics-and-green-ai/)
- [Dettmers et al., LLM.int8(): 8-bit matrix multiplication for transformers at scale (2022)](https://arxiv.org/abs/2208.07339?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode)
- [UNESCO, model compression such as quantisation can save up to 44% of energy](https://www.unesco.org/en/articles/ai-large-language-models-new-report-shows-small-changes-can-reduce-energy-use-90?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode)
- [Hugging Face, Transformers: Optimizing inference, quantization (note: may add latency when not memory-bound)](https://huggingface.co/docs/transformers/main/en/llm_optims?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode)

[All GreenCode Tech Tips](https://greencode.ai/tech-tips/)

#### Key Facts

- Tip **#32**
- For **CTOs and technical leads**
- Category **AI and machine learning**
- Format **short video**
![GreenCode](https://greencode.ai/wp-content/uploads/2024/12/greencode-logo-website-small-colours-2.webp)

Want this checked on your own system? Talk to the GreenCode team.

[Get in touch](https://greencode.ai/contact/)
