Running a 35B Model on a GTX 1080

How to run Qwen3.6 35B on hardware that’s probably gathering dust somewhere.

Prompting the llama-ui within LibreWolf browser

Background

This video Running a 35B AI Model on 6GB VRAM by Codacus piqued my interests, as I’ve been trying out local models with LM Studio on a M4. I had to see if I could replicate their results on my old gaming PC.

I’m getting around a ~20 token/second response rate on hardware that’s about to celebrate its 10th birthday. 🎂

Specs

In 2026, everything about this PC screams, “Please, drop me off at the closest electronics recycling center!” ♻️

We’re going to use it for local model inference. 😈

Cost

The UPS is reporting ~70W when idle, and ~230W under load. Operational cost I’ll estimate to be 0.2kW * 8h * $0.15kWh, $0.24/day, $5.28/month, assuming its not working weekends. I’ll have to pull up some old receipts to know what this rig cost when I built it oh so long ago. Cosidering its age, and seeing as it is pulling double-duty running old FPSes and performing local model inference, I think we can safely call that line item negligible.

The remainder of the information here documents the steps on how to replicate these results on similar hardware.

OS and Drivers

Because CUDA drops support for Compute Capability 6.1 GPUs after CUDA v12.6, and Ubuntu wants to install nvidia-driver-580 by default, we have to do a bit of a dance with apt in order to get our dependencies aligned.

  1. Fall-back to the nouveau video drivers: Software & Updates -> Additional Drivers -> GeForce 1080 -> Using X.Org X server -- Nouveau display driver... -> Apply Changes. Then, reboot.
  2. dpkg -l | grep nvidia and sudo apt purge the listed dependencies. Reboot.
  3. Install the cuda-keyring, and then the nvidia-driver-pinning-570.211.01, cuda-drivers-570 packages.
  4. Pin the driver version, so that apt won’t update it past major version 570.
  5. Install the CUDA Toolkit v12.6

Build llama-cpp with turboquant

$ git clone git@github.com:TheTom/llama-cpp-turboquant
$ cd llama-cpp-turboquant
# specify CUDACXX location, or add the path to nvcc to $PATH
$ export CUDACXX=/usr/local/cuda/bin/nvcc
$ cmake -S . -B build \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=61
$ cmake --build build --config Release -j $(nproc)

Install hf CLI and models

$ curl -LsSf https://hf.co/cli/install.sh | bash
$ hf --install-completion
$ export HF_TOKEN=... # to auth w/ huggingface and enable faster downloads
$ hf download hf://bartowski/Qwen_Qwen3.6-35B-A3B-GGUF/Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf

Run llama-server

# from llama-cpp-turboquant/build
$ ./bin/llama-server \
    -hf bartowski/Qwen_Qwen3.6-35B-A3B-GGUF:Q4_K_M \
    --host 0.0.0.0 \
    --port 8080 \
    --cache-type-k turbo4 \
    --cache-type-v turbo3 \
    --load-mode mlock \
    --cpu-moe \
    --fit-ctx 262144 \
    --fit-target 100

Open localhost:8080, and prompt away! 🤖

Finally…