---
title: "Tech Tip #59: Batch your inference"
description: "An idle GPU waiting for one request at a time wastes power."
url: https://greencode.ai/tech-tips/59-batch-inference-requests/
---

# Tech Tip #59: Batch your inference

**An idle GPU waiting for one request at a time wastes power.**

[Watch the video](https://media.greencode.ai/shorts/batch-inference-requests-32a6aa7bdc29.mp4)

Video created by [explainapaper.com](https://explainapaper.com/?utm_source=greencode.ai&utm_medium=film&utm_campaign=made-by)

### The problem

Serving a model one request at a time leaves the accelerator underused while it still draws substantial power. Energy per answer depends heavily on utilisation.

### The fix you can use today

- Batch concurrent requests with a serving engine built for it.
- Queue non-interactive inference and process it in batches.
- Cap generated tokens to what the task needs.

GreenCode researchers found batching among the easiest ways to save energy when serving an AI model.

### Tools that help

- [vLLM](https://docs.vllm.ai/?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): an LLM serving engine that batches concurrent requests to keep the accelerator busy
- [nvidia-smi](https://developer.nvidia.com/system-management-interface?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): reports the power draw of NVIDIA GPUs from the card's own sensors
- [CodeCarbon](https://github.com/mlco2/codecarbon?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): a Python package that estimates the energy and CO2 of a run from hardware power and regional grid intensity

### How GreenCode helps

GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, each request sent to the smallest part that can do it, and heavy jobs held for cleaner power. [More on this](https://greencode.ai/feature/green-optimised-generative-ai/).

### Tags

#llm#inference#gpu#performance

### Sources

- [GreenCode State of the Art Review, D2.1 (2025), PDF p. 53: batching and request shaping improve accelerator utilisation; serving configurations that minimise unnecessary token generation](https://greencode.ai/publications/d2-1-state-of-the-art-review/)
- [GreenCode State of the Art Review, D2.1 (2025), PDF p. 39: inference energy strongly driven by runtime and GPU-hours at power](https://greencode.ai/publications/d2-1-state-of-the-art-review/)
- [Hugging Face, Transformers: Optimizing inference, continuous batching](https://huggingface.co/docs/transformers/main/en/llm_optims?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode)

[All GreenCode Tech Tips](https://greencode.ai/tech-tips/)

#### Key Facts

- Tip **#59**
- For **Ops and infrastructure**
- Category **AI and machine learning**
- Format **short video**
![GreenCode](https://greencode.ai/wp-content/uploads/2024/12/greencode-logo-website-small-colours-2.webp)

Want this checked on your own system? Talk to the GreenCode team.

[Get in touch](https://greencode.ai/contact/)
