---
title: "Tech Tip #59: Batch your inference (video)"
description: "An idle GPU waiting for one request at a time wastes power."
url: https://greencode.ai/shorts/59-batch-inference-requests/
---

# Tech Tip #59: Batch your inference (video)

## Tech Tip #59: Batch your inference

**An idle GPU waiting for one request at a time wastes power.**

[Watch the video](https://media.greencode.ai/shorts/batch-inference-requests-32a6aa7bdc29.mp4)

Video created by [explainapaper.com](https://explainapaper.com/?utm_source=greencode.ai&utm_medium=film&utm_campaign=made-by)

### The problem

Serving a model one request at a time leaves the accelerator underused while it still draws substantial power. Energy per answer depends heavily on utilisation.

### The fix you can use today

- Batch concurrent requests with a serving engine built for it.
- Queue non-interactive inference and process it in batches.
- Cap generated tokens to what the task needs.

GreenCode researchers found batching among the easiest ways to save energy when serving an AI model.

### Tools that help

- [vLLM](https://docs.vllm.ai/?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): an LLM serving engine that batches concurrent requests to keep the accelerator busy
- [nvidia-smi](https://developer.nvidia.com/system-management-interface?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): reports the power draw of NVIDIA GPUs from the card's own sensors
- [CodeCarbon](https://github.com/mlco2/codecarbon?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode): a Python package that estimates the energy and CO2 of a run from hardware power and regional grid intensity

### How GreenCode helps

GreenCode helps tackle this by keeping its own AI lean: small fine-tuned models, each request sent to the smallest part that can do it, and heavy jobs held for cleaner power. [More on this](https://greencode.ai/feature/green-optimised-generative-ai/).

### Tags

#llm#inference#gpu#performance

### Sources

- [GreenCode State of the Art Review, D2.1 (2025), PDF p. 53: batching and request shaping improve accelerator utilisation; serving configurations that minimise unnecessary token generation](https://greencode.ai/publications/d2-1-state-of-the-art-review/)
- [GreenCode State of the Art Review, D2.1 (2025), PDF p. 39: inference energy strongly driven by runtime and GPU-hours at power](https://greencode.ai/publications/d2-1-state-of-the-art-review/)
- [Hugging Face, Transformers: Optimizing inference, continuous batching](https://huggingface.co/docs/transformers/main/en/llm_optims?utm_source=greencode.ai&utm_medium=referral&utm_campaign=greencode)

### Transcript

An AI chip waiting for one request at a time wastes power. Many teams serve a model one request at a time. The chip sits half empty. But it still draws a lot of power. So the energy per answer depends on how busy you keep it. Here is the fix you can use today. First, batch requests that arrive together. Use a serving engine built for that, such as vLLM. Second, queue work that nobody is waiting for, and run it in batches. Third, cap the length of answers to what the task needs. Fewer words generated means less work for the chip. Check the gain, using tools like nvidia-smi and CodeCarbon. GreenCode researchers found batching among the easiest energy savings for serving AI. GreenCode helps tackle this by keeping its own AI lean. Requests go to the smallest part that works, and heavy jobs wait for cleaner power. That was GreenCode Tech Tip number fifty-nine. Follow for the next tip, and find more on greencode.ai/tech-tips.

[The tip in full](https://greencode.ai/tech-tips/59-batch-inference-requests/)

[« Tech Tip #58: Keep hardware in service longer](https://greencode.ai/shorts/58-keep-hardware-longer/)[Tech Tip #60: Does dark mode save battery? »](https://greencode.ai/shorts/60-dark-mode-myth/)

[All GreenCode Tech Tips](https://greencode.ai/tech-tips/) · [All GreenCode videos](https://greencode.ai/videos/)

#### Key Facts

- Tip **#59**
- For **Ops and infrastructure**
- Length **1:06**
- Category **AI and machine learning**
![GreenCode](https://greencode.ai/wp-content/uploads/2024/12/greencode-logo-website-small-colours-2.webp)

Want this checked on your own system? Talk to the GreenCode team.

[Get in touch](https://greencode.ai/contact/)
