HMMs are much more similar to S3, S4, and other deep state-space models than to transformers.

In fact, HMMs are discrete shallow state-space models.

I believe S4 and successors might become serious contenders to transformers.

For a detailed tutorial, see https://srush.github.io/annotated-s4