Of course, if the software or hardware on your phone is compromised, all is lost. This is not specific to Signal though.

> I'm saying that we have to trust them that they don't.

This applies to anything. It is not feasible to build and/or check all software and hardware by yourself.