Default Robots.txt Handler

[paganotoni]

One small addition to the buffalo framework that I would like to do is have the buffalo apps handle robots.txt by default.

How will this work

We will add a default robots handler to buffalo apps, what this handler will do is will render a file in robots.txt that is inside /assets/ folder.

If the file is not present it will render a permissive robots.txt content

User-agent: *
Disallow:

We should also allow developers to change this handler for any other, something like:

app.RobotsHandler = otherRobotHandler

What needs to change

• Need to generate a default robots.txt inside assets folder.
• Need to add a default robots.txt handler that will render the file from the assetsBox.
• Need to add documentation of all this on gobuffalo.io

[u007]

would be better a way to render any files in public/ if it exists without the need to add prefix assets

and also anyway to off directory indexing for app.ServeFiles ?

[markbates]

@paganotoni I like @u007 idea of allowing for the public directory to be served up easily. #853 addresses this.

[u007]

thank you :)