From 398b0af22200827771a81b8757625f4c9a6a248b Mon Sep 17 00:00:00 2001 From: Aaron Ball Date: Wed, 6 Sep 2023 15:57:44 -0600 Subject: Add robots.txt This should stop amazonbot from trying to download every single snapshot of every single ref of every single repo hosted by cgit. :facepalm: --- robots.txt | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 robots.txt diff --git a/robots.txt b/robots.txt new file mode 100644 index 0000000..3f4b679 --- /dev/null +++ b/robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Disallow: /src/*/snapshot/*.tar.gz -- cgit v1.2.3