If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks.
-
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
@osm_tech Interesting. Perhaps we could follow-up via e-mail or DM? alexander.fanta@ftm.nl
-
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
@osm_tech Wow. TIL that software development kits are more or less silently embedding internet scrapers in (unrelated) end-user applications to distribute AI data scraping across residential addresses and therefore be harder to defend against.
Hey, Tim, were you expecting stuff like this 35 years down the line? -
@BalooUriza We use fail2ban to handle some of this with custom rules, but eventually fail2ban becomes a bottleneck after 100,000 IP addresses.
@osm_tech @BalooUriza is it using ipset hashsets, or default rule-per-ip rules? raw namespace or? I don't know the details of implementation, but if it is L7 load that is problematic (instead of pure bandwidth DDoS), it might be worth to consider whitelisting instead. I.e. whitelist addresses (or /24s) that have *not* had excessive requests lately, and put them in priority network bucket, and the rest (which is not blacklisted) goes in best-effort bucket (to maybe migrate to whitelist later)
-
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
-
@osm_tech @BalooUriza For IPv4, a bitmask of the entire address space is a viable "efficient" implementation of blocking. I wonder if there are tools that can do it that way rather than needing a gigantic list.
@dalias @BalooUriza But that is one of the points @osm_tech are making in their post. These crawlers resort to using massive amounts of "scrapers hiding behind residential proxy/embedded-SDK networks" - meaning they are using Adware-infested phones all over the world for their scraping attaks. So banning IP ranges won't help much. Playing cat-and-mouse with these scrapers is resource intensive, which is increasingly hard for FOSS projects and is also driving up cost for commercial offerings.
-
@dalias @BalooUriza But that is one of the points @osm_tech are making in their post. These crawlers resort to using massive amounts of "scrapers hiding behind residential proxy/embedded-SDK networks" - meaning they are using Adware-infested phones all over the world for their scraping attaks. So banning IP ranges won't help much. Playing cat-and-mouse with these scrapers is resource intensive, which is increasingly hard for FOSS projects and is also driving up cost for commercial offerings.
@magezwitscher @BalooUriza @osm_tech Not ranges. Just the single IP, and a short-lived ban. All you need to do is get them down from thousands of requests per minute to one request per hour (because they get banned for an hour each time they start again).
-
@Mimesatwork @osm_tech The same people this message reached for a start.
Journalists no longer have the reach you think they do. They have become extremely unreliable.
Write the post, spread it the same way they spread their request for a journo...
They got at least you and I and the person who shared it with me initially so... they have some reach, especially into the people who care about this kind of thing.
-
@osm_tech Pinging @GarretSidzaka as he might have some leads.
@mrgrumpymonkey @osm_tech
Brian Krebs is on it -
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
@osm_tech hey, @arstechnica it is not only @wikipedia that suffers
-
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
@osm_tech Good luck finding a journalist in 2026.
-
@osm_tech You are definitely not alone: https://lwn.net/Articles/1008897/ The situation is not sustainable but I'm not sure what we do about it beyond waiting for the AI bubble to burst.
-
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
-
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
@osm_tech Heise has written about it in German/English.
-
I feel for yall. These residential proxies and the sdk networks are the bane of my existence and I’m paid to deal with them.
@ThatPrilla @osm_tech It would be very useful with a tool running on ISP hardware that could detect residential proxies. Or is there anything we can stuff into our DNS'es to blackhole the proxies backbone?
-
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
@osm_tech Ping @404mediaco -
E eyjala@mastodon.social shared this topic
-
If you write about the messy reality behind "free" internet services: we're seeing #OpenStreetMap hammered by scrapers hiding behind residential proxy/embedded-SDK networks. We're a volunteer-run service and the costs are real. We'd love to talk to a journalist about what we're seeing + how we're responding. #AI #Bots #Abuse
@osm_tech might be a thing @davidgerard could do on pivot
-
@osm_tech might be a thing @davidgerard could do on pivot
@froztbyte @osm_tech yeah i'm getting the same AI assholes
as is @RationalWiki (i'm the sysadmin trying to keep the site up in the face of the hammering - we can either lose Google search listing, or we can be literally unusable for humans)
as is @corbet at Linux Weekly News - OSM might be relevant to LWN, a free content project getting hammered by the AI bots
they botnet suburban Android boxes
covered it a bit previously on Pivot:
https://pivot-to-ai.com/2025/06/02/fighting-the-ai-scraper-bots-at-pivot-to-ai-and-rationalwiki/
https://pivot-to-ai.com/2025/09/07/the-ai-scraper-bots-are-hammering-pivot-to-ai-again-please-test/ -
@osm_tech Tell me more. You can reach me at sjvn01 <at> gmail.com
-
B bugspriet@social.tchncs.de shared this topic
