Ternary Nemotron 30B-A3B GGUF runs on 12GB GPU with 368K context

Extreme quantization lets a 30B MoE with huge context run on consumer GPUs.

A ternary (1.58-bit) 30B-A3B MoE model, quantized to GGUF, runs on a 12GB GPU with 368K context, 172 tok/s decode and 7000+ tok/s prefill in both stock llama.cpp and the PrismML fork. Nvidia also released a Nemotron-Personas-Germany dataset.

Name

Ternary-Nemotron-3.5-Lightning-30B-A3B

Params

30B (A3B MoE, 1.58-bit ternary)

Context

368K

Domain

general

Read it at Reddit →

Coverage: 2 sources, in order

1. 26 Oct 07:49 Reddit Read the article →
2. 26 Oct 19:00 Hugging Face Read the article →

← Front page