Why robots.txt?

[Zwyx]

Hi,

The docs mention that You must add a robots.txt file to allow search engines to crawl all your application pages.

Why is that?

A robots.txt file allowing everything seems to be unnecessary:

Do I have to include an allow rule to allow crawling?
No, you do not need to include an allow rule. All URLs are implicitly allowed and the allow rule is used to override disallow rules in the same robots.txt file.

Also:

• https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
• https://en.wikipedia.org/wiki/Robots.txt#Examples

Thank you!

[joshas]

Probably a measure to keep PageSpeed Insights happy and allow you to get that perfect 100 in "SEO" category?

[Zwyx]

Thanks for your reply Joshas, but that's not the reason: I have 100 for SEO, without any robots.txt file.