Not only that, but if it's legal not to do business with a minority group, then people who aren't themselves prejudiced will come under pressure not to do business with or employ members of the minority group in order to keep the prejudiced people who don't want to associate with those minorities happy.