The point is SRP is no different, in this context, from storing a standard password hash. You can brute force the passcode if you have access to the verification material on the server.
Apple's device security model relies on rate- and attempt-limiting unlock attempts, so that people can in fact use short numeric passcodes. Their on-device model is carefully designed to make it very hard to bypass this.
The problem is now they have extended that model and attack surface to their HSM clusters. That's 1) not Apple hardware (I trust Apple hardware more than I trust the third-party HSMs they use), 2) not (only) Apple code (again I have less trust in HSM frameworks than Apple's), 3) shared for many users, so break one HSM cluster and you get to bruteforce a lot of passcodes, 4) strangely not documented in Apple's Platform Security Guide, which is very suspicious.
Apple's device security model relies on rate- and attempt-limiting unlock attempts, so that people can in fact use short numeric passcodes. Their on-device model is carefully designed to make it very hard to bypass this.
The problem is now they have extended that model and attack surface to their HSM clusters. That's 1) not Apple hardware (I trust Apple hardware more than I trust the third-party HSMs they use), 2) not (only) Apple code (again I have less trust in HSM frameworks than Apple's), 3) shared for many users, so break one HSM cluster and you get to bruteforce a lot of passcodes, 4) strangely not documented in Apple's Platform Security Guide, which is very suspicious.