Recovering 4.1 GiB of RAM on NVIDIA DGX Spark
Wouldn't you like your KV pool to grow from 262,144 to 937,984 tokens—3.58× the capacity—with about 4 GiB more memory per host?
Read articleMarcin Junczys-Dowmunt
I work with models and language, and sometimes even with language models. And just for the heck of it, I am trying to do it all in C++.
This site sits somewhere between professional work and hobby projects, which is what happens to people who have no real concept of work-life balance.
Marian: neural machine translation in C++.
Wouldn't you like your KV pool to grow from 262,144 to 937,984 tokens—3.58× the capacity—with about 4 GiB more memory per host?
Read articleIf you suspect that significant parts of this were written by and with AI, then you are absolutely right.