We had a state management and deployment system through which all changes were effected that would automatically rollback changes if the smoke test failed, or if one of the ops staff found an issue.