Can rank decomposition be used to reduce the original weight matrices as well? O... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

seydor on March 24, 2023 | parent | context | favorite | on: LoRA: Low-Rank Adaptation of Large Language Models

Can rank decomposition be used to reduce the original weight matrices as well? Or are they assumed to be compressed already?

metanonsense on March 29, 2023 [–]

Those fully trained networks are usually considered full-rank. At least that is what they say in the LoRA paper.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact