llama.cpp PR adds LRFU-style hybrid MoE expert cache

A pull request for llama.cpp replaces the SLRU expert cache with a hybrid cache that ranks frequently used experts via decaying usage history. With a warm cache it reports about +6% decode on QFN and +7% on GLM versus SLRU with 8% fewer uploads.

Domain

general

Read it at Reddit →

Coverage: 1 sources, in order

1. 06 Nov 15:39 Reddit Read the article →

← Front page