Quantum mechanics: frustrating formal mathematicians for over a century.
You'd have to exercise caution when using (quasi-)probabilities less than zero or greater than one, because many of the classical properties of probability theory depend upon those conditions.
