This is quite a different scenario, because the model has explicitly been trained to be polite, so as to avoid journalists and others that benefit from alarmism hopping on and getting it to say mean things and writing articles about how the AI was mean to me.
If you want to make it analogous to self-driving, it is like training the car to always give way to pedestrians and someone "defeating" the technology by saying, look I can stand in front of the car and it won't move forward.
This is a valid point, but the “correct” behavior in that instance should be to communicate the constraint, instead of claiming that it would be followed. This is certainly more innocuous than most confident BS-ing that these models do, but it’s still an instance of it. In the analogy, it’d be like standing in front of the car and having it stop, but still tell the passengers that it’s moving at 60mph. The constraint itself isn’t the problem
If you want to make it analogous to self-driving, it is like training the car to always give way to pedestrians and someone "defeating" the technology by saying, look I can stand in front of the car and it won't move forward.