To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
-
@JonSaenzAgirre It is a good questions, and we don't know the answer either. Our planet data is so much easier to process and use.
@osm_tech @JonSaenzAgirre thats dumb ai, probably. No "i" at all...
-
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
#AI #Bots #Abuse -
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
#AI #Bots #Abuse@osm_tech sounds familiar, last year I braved turning cloudflares "under attack" mode off for https://dnshistory.org/ and saw an extra 5 million requests/day (500k unique IPs) overloading things. It's still blocking >700k requests/day a month later...
-
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
#AI #Bots #Abuse@osm_tech and we can tell the scrapers are AI built because a cursory glance at the documentation on the "coders" part would've prevented this problem.
-
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
#AI #Bots #Abuse@osm_tech Thank you. I'm a beginner who has just been doing toy projects and has barely any notion of what web scraping is but I'm very happy to learn that your data can be downloaded
-
@osm_tech I wonder if the culprit will ever come forward, apologise, and change their ways? Someone tasked these proxy scrapers with ridiculous requests.
Have they been targeting the main OSM API, the website interface designed for humans, or Overpass? -
@osm_tech does coming from residential IPs mean that someone has baked a scraper into some popular tool that people don't realize is doing that?
-
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
#AI #Bots #Abuse@osm_tech Limit the speed to Modem 14400 speed each IP for a month or so.
-
N necrosis@chaos.social shared this topic
-
@osm_tech question. Why do people scrape server which make the data freely available? And, probably, better structured in the final product. I don't see the point.
@JonSaenzAgirre @osm_tech
The scrapers are DUMB.
They are not curated, have only basic maintenance, are built to gobble up ANYTHING textual they encounter, without respect, mercy or reason.Just collect meaningless data.
That’s the nature of the coveted LLMs: just statistics, no understanding, structure or meaning.
And greedy crooks in haste to make quick money just grab everything they can.
The AI bubble needs to pop really soon.
-
@osm_tech and we can tell the scrapers are AI built because a cursory glance at the documentation on the "coders" part would've prevented this problem.
@ClariNerd @osm_tech Because their IP ranges are increasingly being blocked by servers following their harmful scraping habits, AI companies are now releasing "browsers" so they can scrape from residential IPs instead and circumvent blocks. Oh, sorry, I meant "so they can empower users with AI insight in this new era of information".
-
@ClariNerd @osm_tech Because their IP ranges are increasingly being blocked by servers following their harmful scraping habits, AI companies are now releasing "browsers" so they can scrape from residential IPs instead and circumvent blocks. Oh, sorry, I meant "so they can empower users with AI insight in this new era of information".
-
A angelacarstensen@mastodon.online shared this topic
