Not sure what you mean by complicated API. The (pylab) API is a very straightforward (mostly) immediate rendering typeish interface, with a lot of convenient shortcuts for operations used a lot in data analysis.
For architecture astronauts there's also the OOP API over which the pylab API is a wrapper.
Of course there are also a lot of all sorts of declarative APIs, which are popular with people copy-pasting code from cookbooks. These become very painful very fast if you do something that's not in the cookbook.
Matplotlib does struggle with performance in some/many cases, but it has little to do with the API.
Just my personal experience from using the library for at least 7-8 years. So many things and concepts are glued onto each other, making the API so much non-intuitive whenever you try to do anything more sophisticated that isn't a 1:1 match from examples found in the cookbook. It's really a PITA and performance, I have to say this again, is really really bad. If this had been part of my daily job I would certainly try to switch to something else.
Performance for animation and (custom) interaction is a real problem. But as for performance being "really really bad", there are not many widely used plotting libraries faster than it, at least for static plotting and zoom/pan interaction.
There are indeed many ad-hoc functions, typically for commonly used cases, and they tend to cover vast majority of common use case with very simple and concise code. If you want something more custom, the underlying artist API is very flexible. But you probably know this based on the 7-8 years?
Things like subplot layouts, data point annotation and legend tweaking can be really painful. Something like a proper box/model CSS layouting would be great.
I feel like everyone has different expectations for a scientific plotting API. The tension between ease of use and expressivity is so strong that a one-size-fits-all solution is unlikely ever to exist.
of which the matplotlib is the embodiment. Terrible API with terrible terrible performance.