302fa154f4
This reverts commit d30566d8f6.
This is needed by Lighthouse to fetch robots.txt and it's worth making a
harmless exception for it to work properly.
https://github.com/GoogleChrome/lighthouse/issues/4386