Add robots.txt

- To prevent files/ from being indexed by spiders
4 years ago · ff4a7dd066
parent 6b5470229d
commit ff4a7dd066
1 changed files with 8 additions and 0 deletions
--- a/robots.txt
+++ b/robots.txt
@ -0,0 +1,8 @@
+User-agent: *
+Disallow: /files/
+
+User-agent: *
+Allow: *
+
+#Sitemap: https://www.aney.co.uk/sitemap.xml
+