more parameters at low-precision is better than less parameters at high-precision

https://x.com/i/status/2039207501477097541

Reasoning ability comes from the structure of billions of connections, not the precision of each one. an 8b model with coarse weights still captures more of the original model's knowledge than a small model with precise weights.

The AI Race

Frontier intelligence in the West has become all about extrapolation and scaling laws. Meanwhile, those who are compute-constrained are left to gather the crumbs. I am incredibly grateful to Chinese AI companies for open-sourcing their models. It feels like the world has turned upside down in the tech race: the US used to lead in open-source tech, but now they have doubled down on proprietary solutions.

Geopolitics has divided the world of technology that once united us. Maybe it is time to adapt to the tech ecosystem created by our neighbor to the north...