It all depends on particular design. There are even some out there where baseband OS and Linux run under the same proprietary hypervisor. On modern phones it's pretty rare to be able to say with 100% certainty that the baseband OS doesn't have any access to user memory - and often you can say with 100% certainty that it can access other interesting stuff, like GPS or microphone, without any supervision from the user.