Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How do you circumvent cloud provider IP blocks? For example, one site blocks all requests from AWS EC2 servers.


None of the sites I'm scraping do that, mostly.

I'm not scraping high value sites like that (I mostly target amateur original content). It's not really of interest to other businesses. As such, I tend to just run into things like normal cloud-flare wrapped sites, and one place that tried to detect bots and return intentionally garbled data.

If I run into that sort of thing, I guess we'll see.


I've never used this, and it's incredibly shady considering the users probably do not realize that their Hola browser plugin does this, but Hola runs a paid VPN service where you can get thousands of low-bandwidth connections on unique residential IP addresses, provided generously through their "free" VPN users.... It's essentially a legitimate attempt at running a botnet as a service.

But if the end justifies the means... http://luminati.io/




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: