# pjtr.be — development host for the Add Some Spite shop. # # Crawling is ALLOWED on purpose. Keeping this host out of search is done by # the "X-Robots-Tag: noindex" header in .htaccess, and a crawler has to be able # to FETCH a page to see that header. Blocking here with "Disallow: /" would # stop the fetch, so the pages already in the index would never be removed. # # Switch to "Disallow: /" only once Search Console shows no pjtr.be URLs left. # Full reasoning: BUILD.md, "Keeping pjtr.be out of search engines". # # NOT FOR THE PRODUCTION DOMAIN — replace this file there. User-agent: * Allow: /