Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

By abandoning their AJAX crawling scheme as described in the OP, they are essentially saying that they will evaluate JS for all sites. Do you have some reason to doubt that?


If crawling with JS costs 1,000X or 10,000X as much as crawling without, it's fair to say that even Google isn't going to crawl 100s of billions of pages executing JS.

As a former web-scale search engine CTO, my opinions are commonly surprising to folks who haven't built a web-scale crawler/search engine.


My own experiments/experience shows that recrawls happen about 1/3 as often and tend to lag a few days behind for JS content vs inlined/delivered content. It's helped a little by dynamically delivering the sitemap data, but even that only speeds things up a little.

My guess is they're putting about 1/10th the effort into keeping things freshly indexed for JS, but may well be devoting 2x the resources vs directly received content.


> By abandoning their AJAX crawling scheme as described in the OP, they are essentially saying that they will evaluate JS for all sites.

No, they are not. If you even think that's possible you're fundamentally misunderstanding how search engines work.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: