Monitor robots.txt noindex/nofollow

Monitor that noindex and nofollow are not set in robots.txt files on production environments. This can severely harm search results.

This prevents mistakes, e.g. when deploying from a staging environment on which these options are rightfully set.