robots.txt and ai.txt ignored #19458
Schamschula
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
As a webmaster, I've found the agent string
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0 (compatible; pytorch-lightning; +https://github.com/Lightning-AI/pytorch-lightning)"
for requests in my logs. However, there never was a corresponding request from the host torobots.txt
orai.txt
. Even worse, there are requests by"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0"
without the link to GitHub.com (perhaps an older version?). Unless I know what the data is used for I will block such agents from my site as scraping of graphics is not appreciated.Beta Was this translation helpful? Give feedback.
All reactions