IANAL, but I think that there is a bit of an exception in the legal doctrine of promissory estoppel (http://en.wikipedia.org/wiki/Estoppel#Promissory_estoppel), which says that one cannot withdraw a promise to another party that they have come to rely upon. If that promise is implied by inaction, it is still sometimes seen as a promise.
You're thinking of trademarks. Patents can't become unenforceable thru neglect... trademarks, though, can...