If you want to not quantize at all, you need to double it for fp16—16GB. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

lolinder on April 18, 2024 | parent | context | favorite | on: Meta Llama 3

If you want to not quantize at all, you need to double it for fp16—16GB.

cjbprime on April 19, 2024 [–]

Yes, but I think it's standard to do inference at q8, not fp16.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact