Hello, A client of mine is going through a bit of a crisis. A developer (at their end) added Disallow: / to the robots.txt file.
Hello Mozzers! I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt.
Old Hard to Find TV Series on DVD
Moz Crawler is not able to access the robots.txt due to server error. Please advice on how to tackle the server error.
I currently have this under my robot txt file: User-agent: * Disallow: /authenticated/ Disallow: /css/ Disallow: /images/ Disallow: /js/
Hi guys, When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.
I am receiving the following error message through webmaster tools http://www.sourcemarketingdirect.com/: Googlebot can't access your site ...
Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.
After a site audit, I find out that my website don't have a robot.txt. Does it hurt my website rankings? One more thing, when I type ...
Moz are reporting the robots.txt file is blocking them from crawling one of our websites. But as far as we can see this file is exactly the ...
I've run our robots.txt file through the tester in Webmaster tools and it is saying allowed, however I'm obviously concerned about the warnings.