For simple scraping where the content is fairly static, or when performance is critical, I will use linkedom to process pages.

https://github.com/WebReflection/linkedom

When the content is complex or involves clicking, Playwright is probably the best tool for the job.

https://github.com/microsoft/playwright