You could make the same argument against the real numbers. Almost all of them cannot even be written down or described, so it's hard to say how they map to the real-world. But the concept of real numbers is extremely useful for a large number of mathematical proofs.
It's not that we rely on such things, but that these things are unavoidable consequences of choices we make that seem to be perfectly reasonable, and result in useful math.
Phrasing in the other way:
As we develop math that we find useful and powerful, we find that we have to make choices. Those choices have consequences, and sometimes as we explore the consequences we find that really strange things happen.
We can go back and make different choices, but in practice we tend to find that no matter what choices we make there are odd and hairy things that result.
