Skip to content

Instantly share code, notes, and snippets.

@miah-oss

miah-oss/robots.sh

Created July 30, 2024 14:28
Show Gist options
  • Save miah-oss/fde332d11d1ea80073cd9806e44a934f to your computer and use it in GitHub Desktop.
Save miah-oss/fde332d11d1ea80073cd9806e44a934f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
robots.sh
#!/bin/sh
set -eu
TOKEN="$ROBOTS_TXT_TOKEN"
curl -vv \
-X "POST" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "@robots_body" \
https://api.darkvisitors.com/robots-txts
Raw
robots_body
{"agent_types":["AI Assistant","AI Data Scraper","AI Search Crawler","Undocumented AI Agent"],"disallow":"/"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment