This reverts commit d30566d8f6597fad50c950fdc7bee1f809445512. This is needed by Lighthouse to fetch robots.txt and it's worth making a harmless exception for it to work properly. https://github.com/GoogleChrome/lighthouse/issues/4386