Anubis uses a javascript-based challenge to stop scrapers from hammering your server.
Installing it is easy enough but then you need to figure out how to configure it and it’s pretty intense. Here is the config I have for piefed.social, so far:
bots:
- name: allow-api-json-xhr
user_agent_regex: .* # Required dummy pattern to pass validation
cel:
expression: >
request.path.startsWith("/api/")
|| ("accept" in request.headers
&& (request.headers["accept"].lower().contains("application/ld+json")
|| request.headers["accept"].lower().contains("application/activity+json")))
|| ("x-requested-with" in request.headers
&& request.headers["x-requested-with"].lower() == "xmlhttprequest")
action: ALLOW
- name: mozilla-user-agents
user_agent_regex: "Mozilla.*"
action: CHALLENGE
challenge:
difficulty: 2
algorithm: fast
report_as: 21
- import: (data)/meta/ai-block-aggressive.yaml
- import: (data)/crawlers/_allow-good.yaml
- import: (data)/common/keep-internet-working.yaml
status_codes:
CHALLENGE: 200
DENY: 200
thresholds:
# By default Anubis ships with the following thresholds:
- name: minimal-suspicion # This client is likely fine, its soul is lighter than a feather
expression: weight <= 0 # a feather weighs zero units
action: ALLOW # Allow the traffic through
# For clients that had some weight reduced through custom rules, give them a
# lightweight challenge.
- name: mild-suspicion
expression:
all:
- weight > 0
- weight < 10
action: CHALLENGE
challenge:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh
algorithm: metarefresh
difficulty: 1
report_as: 1
# For clients that are browser-like but have either gained points from custom rules or
# report as a standard browser.
- name: moderate-suspicion
expression:
all:
- weight >= 10
- weight < 20
action: CHALLENGE
challenge:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
algorithm: fast
difficulty: 2 # two leading zeros, very fast for most clients
report_as: 2
# For clients that are browser like and have gained many points from custom rules
- name: extreme-suspicion
expression: weight >= 20
action: CHALLENGE
challenge:
# https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
algorithm: fast
difficulty: 4
report_as: 4
Sorry about the line wraps. Copy and paste it into a text editor and it’ll be more readable.
The tricky bit is at the start where we try to exclude API requests and ActivityPub requests from the blocking. Anubis has it’s own little language which is very powerful but a real hassle to debug. Hopefully this post saves you some time.