Robots.txt not valid – syntax errors in HTML

General Discussion
2

UPDATE: PROBLEM SOLVED

In case anyone ever runs into the same problem, here is what finally solved the problem:

In the public folder of your project, you’ll need the following files:

_redirects
When deploying SPA’s with client side routing to Netlify, add this code snippet to ensure that the navigation works correctly: /* /index.html 200
And, to make sure that the robots.txt file is not treated as client side routing, add this code snippet too: /robots.txt /public/robots.txt 200

_headers
Add this code snippet:

/public/robots.txt
  Content-Type: text/plain

robots.txt
Generate a robots.txt file including a url to a sitemap file (can be done online using a robots.txt file generator) and add the code here

Without these steps, the robots.txt file in my app was read as HTML, the redirect to index.html kicked in and the Lighthouse report warned about syntax errors in the robots.txt file

2 Likes