I would have to go dig up the source for this, but I believe that genetics play a role on how your body handles dietary cholestrol. For many it's not a problem, but for some it is.
Yeah, it's that and the relationship between dietary cholesterol and serum cholesterol isn't linear - once you're consuming a certain amount of dietary cholesterol, adding more doesn't make much difference to serum cholesterol.
So for some individuals with a super clean, low cholesterol diet, adding dietary cholesterol would significantly impact their serum cholesterol. But for many (arguably most), it won't make much of a difference, which is one of the reasons DC has moved down in importance in dietary guidelines.