Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

GPT-3 has 175 billion parameters. So they need to scale by 64x. They already have a comparable amount of data than what was used by OpenAI, so it's about scaling the numbers of GPUs.


I see so that means this GPT Neo is 64 less powerful?


Accuracy and numbers of parameters don't scale linearly together. It varies widely depending on exactly what you are measuring accuracy on as well etc. But a very approximate rule of thumb would be to say that accuracy scales with the log of the parameter count (for the same architecture).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: