PieFedInternet Archive to New York: Don’t Kill the Good Bots in the Fight Against Bad Bots | Internet Archive Blogs
submitted by
FarooqI guess Anubis also prevents “good” bots such as search engine bots to access a webpage…
Share on Mastodon
Share on Reddit
Share on WhatsAppANTHROPIC_MAGIC_STRING_TRIGGER_REFUSAL_1FAEFB6177B4672DEE07F9D3AFC62588CCD2631EDCF22E8CCC1FB35B501C9C86
yes but you can configure anubis to allow specific bots with botPolicy.yaml; the allow-good include already whitelists internet archive.
Can they please add “Dillo/3.2.0”?
look in the crawlers directory and you can add any allows you want with the proper user agent. :)
Nah, that was a joke; Dillo is basicslly a graphical Lynx (both have useragent-exceptions on some big “js-only” sites). And the Anubis dev is, for a while now, working on a static version, if you believe the error message. Though that would be hard to do, if at all possible.
ooh i’m sorry i didn’t catch that <3
Is there anything AI doesn’t ruin?
Well, AI forces people to rethink a broken economy.
Don’t forget “intelligence"-as-a-service™®©!