review site for html injections #722

chadwhitacre · 2013-03-12T02:50:34Z

I just discovered that I wasn't escaping statements. :-(

I'm not sure how that happened, but it did. We need to review the whole site for html injections.

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

wilkie · 2013-03-12T07:32:41Z

In investigating the issue 8506070 caused, I've audited participant statement usage throughout the site.

User Statement

stored non-escaped

statement.json: wrapped!
index.html: meta og:description: SERIOUSLY BAD: not-escaped (I confirmed in a development environment that I can indeed close the tag and inject a <link> or <meta> etc in <head>, which is obvi-bad)
index.html: displaying the statement: escaped twice! see #723 This doesn't seem to be a place where escape is needed, oddly.
about/stats: escaped for every user
about/goals.html: BAD: not escaped
templates/profile-edit.html: intentionally not escaped

chadwhitacre · 2013-03-14T01:04:05Z

Fixed in b4bdc32: index.html: meta og:description: SERIOUSLY BAD: not-escaped (I confirmed in a development environment that I can indeed close the tag and inject a <link> or <meta> etc in <head>, which is obvi-bad)

Fixed in 959e72a: index.html: displaying the statement: escaped twice! see #723 This doesn't seem to be a place where escape is needed, oddly.

Fixed in cca632d: about/goals.html: BAD: not escaped

Good to close?

wilkie · 2013-03-14T01:10:22Z

well, the issue suggests looking for html injections throughout the site. I only focused on statement fields, so technically the issue is not yet fulfilled.

chadwhitacre · 2013-03-14T03:53:44Z

Good point. :)

@buttscicles

ht @buttscicles

chadwhitacre · 2013-04-03T21:02:49Z

@buttscicles caught a few more of these (thanks!). I'm beginning to realize now why Tornado moved to HTML-escaping by default in 2.x. This feels like whack-a-mole. Gittip uses Aspen's default templating language, Tornado 1.x. We have a ticket over on Aspen to upgrade to 2.x (AspenWeb/pando.py#158). Picking up over there ...

chadwhitacre · 2014-01-05T19:46:40Z

Let's take this up now that we've migrated to Jinja2 (#1783).

chadwhitacre · 2014-01-30T02:53:19Z

@clone1018 Are we good to close now that we're on Jinja2?

clone1018 · 2014-01-30T02:57:52Z

I wouldn't say we're good but we're good for now.

chadwhitacre · 2014-01-30T14:12:59Z

@clone1018 Would you be willing to conduct a review so we can close this ticket?

clone1018 · 2014-01-30T14:14:48Z

@whit537 I wouldn't feel completely safe until we change Jinja2 to being autoescape by default

http://jinja.pocoo.org/docs/extensions/#autoescape-extension

clone1018 · 2014-01-30T14:17:12Z

I'm in favor of closing this specific ticket for now, we can reticket any other issues and if escaping becomes a problem we can ticket the autoescape extension.

chadwhitacre · 2014-01-30T14:23:49Z

I'm leery of autoescape, because the rules for escaping differ with the syntax you're templating into. If we're inserting into an href attribute like href="{{ foo }}" then we have different escaping needs than if we are inserting into an HTML context. Auto-escaping can give us a false sense of security that everything is escaped properly when we really still need to be aware of every single place in templates where we insert stuff and whether or not we're doing it right.

Even with autoescape we still need this audit.

seanlinsley · 2014-01-30T14:44:41Z

Could you go into detail @whit537 about the sorts of escaping rules that could cause trouble? In general, automatically escaping everything is a boon to security because it gives you a much smaller surface area to check for holes.

chadwhitacre · 2014-01-30T14:48:43Z

https://www.owasp.org/index.php/XSS_(Cross_Site_Scripting)_Prevention_Cheat_Sheet

seanlinsley · 2014-01-30T14:50:50Z

So you're saying... it might not use the right type of escaping for the given context?

chadwhitacre · 2014-01-30T14:52:54Z

Yup. We got bit by this once already, iirc.

chadwhitacre · 2014-01-30T14:54:12Z

Basically, we need an audit based on that cheat sheet. Whether or not we use autoescaping is orthogonal.

chadwhitacre · 2014-01-30T15:29:40Z

Really, it would be great to have some tooling that presented a list of all the places in templates where we insert stuff, so we can review and check them off one by one.

chadwhitacre · 2014-11-03T21:02:52Z

Now that we have i18n we should consider including that in a review. IRC

blrhc · 2014-11-11T20:20:01Z

I'll start taking a look at this. Thanks.

chadwhitacre · 2014-11-12T13:42:25Z

IRC

chadwhitacre · 2015-02-04T17:53:47Z

Retort:

We're only using Jinja2 for HTML.
But it doesn't hurt to use it when it's not needed.
Hmmm ... really?

chadwhitacre · 2015-02-04T17:58:31Z

It just seems increasingly stupid to rely on "a human knows" for the 544 places where we are potentially vulnerable to injection. Performance will have to be really bad to justify not switching to autoescaping. No?

chadwhitacre · 2015-02-04T18:06:11Z

http://jinja.pocoo.org/docs/dev/api/#autoescaping

chadwhitacre · 2015-02-04T18:16:41Z

Autoescape interacts with i18n somehow:

chadwhitacre · 2015-02-04T18:18:43Z

Also if [newstyle gettext calls] are used in an autoescaping environment they better support automatic escaping.

http://jinja.pocoo.org/docs/dev/extensions/#newstyle-gettext

chadwhitacre · 2015-02-04T18:20:18Z

So it looks like implementing autoescaping will take some work. It's not a trivial quick answer to this ticket.

Changaco · 2015-02-04T18:23:07Z

We're only using Jinja2 for HTML.

Not true, we also use it for plain text in email simplates.

chadwhitacre · 2015-02-04T18:24:52Z

P.S. The place to implement autoescaping would be in aspen_jinja2, by adding autoescape=True, extensions['jinja2.ext.autoescape'] to the Environment instantiation. Then, another thing to work around is that we don't want to autoescape the CSS and JS assets. The workaround for that is:

[---]
{% autoescape false %}{{ css }}{% endautoescape %}

If the application enables the Autoescape Extension one can activate and deactivate the autoescaping from within the templates.

http://jinja.pocoo.org/docs/dev/templates/#autoescape-extension

chadwhitacre · 2015-02-04T18:26:39Z

Not true, we also use it for plain text in email simplates.

Okay, so if we go with autoescaping I guess we'd use the {% autoescape false %} mechanism to turn it off for the text/plain email parts.

chadwhitacre · 2015-02-04T18:27:12Z

@Changaco Can you speak to whether we're using newstyle gettext calls?

Changaco · 2015-02-04T18:33:16Z

Sort of, we use custom functions.

Figured this would be worth having for now until we switch to autoescape (#3152).

chadwhitacre · 2015-02-04T22:53:36Z

Here's a slightly differently-formatted list:

https://gist.github.com/whit537/03b1e5f88fcf38c3dd87

Looks like we have 527 replacements on master right now.

chadwhitacre · 2015-02-05T15:57:34Z

I separated out the places where we are just doing i18n ({{ _("Greetings, program!") }}).

https://gist.github.com/whit537/007d96927377a7ed0419

chadwhitacre · 2015-02-05T16:14:16Z

Category	N
High Risk	336
Simple i18n	139
Assets	23
Escaped	29

https://gist.github.com/whit537/0a60f06b87ecc332f4fe

chadwhitacre · 2015-02-05T16:16:06Z

Alright, I still don't want to go through 336 of these. :-)

chadwhitacre · 2015-02-05T16:17:58Z

The next step down the autoescaping path is to benchmark performance. If performance isn't "really a lot" worse with autoescaping then we should just use that. It'll take a chunk of work to switch over though due to i18n so benchmarking probably makes sense to make sure it'll be worth it.

chadwhitacre · 2015-02-05T16:18:31Z

If performance is really as atrocious as promised then we should review the 336 places and make it part of our PR review habit.

chadwhitacre · 2015-02-05T19:06:33Z

Here's a script I wrote to help with benchmarking:

https://gist.github.com/whit537/352a6b02d5f644720007

Gonna do some benchmarks now.

chadwhitacre · 2015-02-05T19:10:37Z

[gratipay] $ honcho run -e defaults.env,local.env ./env/bin/python benchmark-autoescaping.py | grep " seconds "
11.9522500038 seconds without autoescaping
12.4190990925 seconds *with* autoescaping
[gratipay] $

#!/usr/bin/env python
"""This is a helper script for benchmarking page performance re: #722.

Set this in local.env to avoid a canonicalizing redirect:

    CANONICAL_HOST=

Then run this script like so:

    [gratipay] $ honcho run -e defaults.env,local.env ./env/bin/python benchmark-autoescaping.py

"""
import time
import aspen_jinja2_renderer
from gratipay.testing import Harness


def autoescaping_compile_meta(self, configuration):
    loader = None
    if configuration.project_root is not None:
        # Instantiate a loader that will be used to resolve template bases.
        loader = aspen_jinja2_renderer.FileSystemLoader(configuration.project_root)
    return aspen_jinja2_renderer.Environment( loader=loader
                                            , autoescape=True
                                            , extensions=['jinja2.ext.autoescape']
                                             )

start = time.time()
for i in range(100):
    Harness.client.GET('/')
print("{} seconds without autoescaping".format(time.time() - start))

aspen_jinja2_renderer.Factory.compile_meta = autoescaping_compile_meta

start = time.time()
for i in range(100):
    Harness.client.GET('/')
print("{} seconds *with* autoescaping".format(time.time() - start))

chadwhitacre · 2015-02-05T19:11:35Z

10x

[gratipay] $ honcho run -e defaults.env,local.env ./env/bin/python benchmark-autoescaping.py | grep " seconds "
1.3902361393 seconds without autoescaping
1.18036794662 seconds *with* autoescaping
[gratipay] $

chadwhitacre · 2015-02-05T19:27:45Z

[gratipay] $ honcho run -e defaults.env,local.env ./env/bin/python benchmark-autoescaping.py 10 | grep " seconds "
1.29223489761 seconds without autoescaping
1.17796278 seconds *with* autoescaping
[gratipay] $ honcho run -e defaults.env,local.env ./env/bin/python benchmark-autoescaping.py 100 | grep " seconds "
11.9243550301 seconds without autoescaping
12.4741439819 seconds *with* autoescaping
[gratipay] $ honcho run -e defaults.env,local.env ./env/bin/python benchmark-autoescaping.py 1000 | grep " seconds "
124.910935163 seconds without autoescaping
132.204340935 seconds *with* autoescaping
[gratipay] $

#!/usr/bin/env python
"""This is a helper script for benchmarking page performance re: #722.

Set this in local.env to avoid a canonicalizing redirect:

    CANONICAL_HOST=

Then run this script like so:

    [gratipay] $ honcho run -e defaults.env,local.env ./env/bin/python benchmark-autoescaping.py

"""
import sys
import time

import aspen_jinja2_renderer
from gratipay.testing import Harness


def autoescaping_compile_meta(self, configuration):
    loader = None
    if configuration.project_root is not None:
        # Instantiate a loader that will be used to resolve template bases.
        loader = aspen_jinja2_renderer.FileSystemLoader(configuration.project_root)
    return aspen_jinja2_renderer.Environment( loader=loader
                                            , autoescape=True
                                            , extensions=['jinja2.ext.autoescape']
                                             )

try:
    n = int(sys.argv[1])
except IndexError:
    n = 10

start = time.time()
for i in range(n):
    Harness.client.GET('/')
print("{} seconds without autoescaping".format(time.time() - start))

aspen_jinja2_renderer.Factory.compile_meta = autoescaping_compile_meta

start = time.time()
for i in range(n):
    Harness.client.GET('/')
print("{} seconds *with* autoescaping".format(time.time() - start))

chadwhitacre · 2015-02-05T19:29:27Z

Alright, performance concerns look like FUD to me.

wilkie mentioned this issue Mar 12, 2013

Participant statements are html escaped twice #723

Closed

wilkie mentioned this issue Mar 12, 2013

Bugfix/prevent html injects in statements #724

Merged

chadwhitacre mentioned this issue Mar 12, 2013

links in statements are effed up #727

Closed

chadwhitacre added a commit that referenced this issue Apr 3, 2013

Escape {user}names of connected accounts; #722

a4b904f

ht @buttscicles

chadwhitacre mentioned this issue Apr 3, 2013

Please require a current version of Tornado AspenWeb/pando.py#158

Closed

chadwhitacre added a commit that referenced this issue Apr 3, 2013

List @buttscicles with @wilkie; #722

62a9527

ghost assigned chadwhitacre Oct 12, 2013

Changaco added security labels Aug 13, 2014

chadwhitacre mentioned this issue Feb 4, 2015

switch to autoescape #3152

Closed

chadwhitacre added a commit that referenced this issue Feb 4, 2015

Here's a helper for #722

d7c0882

Figured this would be worth having for now until we switch to autoescape (#3152).

chadwhitacre added a commit that referenced this issue Feb 4, 2015

Here's a helper for #722

6f4d85b

Figured this would be worth having for now until we switch to autoescape (#3152).

This was referenced Feb 5, 2015

turn on autoescape in Jinja2 AspenWeb/aspen.py-plugins#9

Closed

turn on autoescape #3159

Closed

Here's a helper for #722 #3153

Closed

turn on autoescape #3164

Merged

Changaco closed this as completed in #3164 Feb 10, 2015

review site for html injections #722

review site for html injections #722

Comments

chadwhitacre commented Mar 12, 2013

wilkie commented Mar 12, 2013

User Statement

chadwhitacre commented Mar 14, 2013

wilkie commented Mar 14, 2013

chadwhitacre commented Mar 14, 2013

chadwhitacre commented Apr 3, 2013

chadwhitacre commented Jan 5, 2014

chadwhitacre commented Jan 30, 2014

clone1018 commented Jan 30, 2014

chadwhitacre commented Jan 30, 2014

clone1018 commented Jan 30, 2014

clone1018 commented Jan 30, 2014

chadwhitacre commented Jan 30, 2014

seanlinsley commented Jan 30, 2014

chadwhitacre commented Jan 30, 2014

seanlinsley commented Jan 30, 2014

chadwhitacre commented Jan 30, 2014

chadwhitacre commented Jan 30, 2014

chadwhitacre commented Jan 30, 2014

chadwhitacre commented Nov 3, 2014

blrhc commented Nov 11, 2014

chadwhitacre commented Nov 12, 2014

chadwhitacre commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

Changaco commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

Changaco commented Feb 4, 2015

chadwhitacre commented Feb 4, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015

chadwhitacre commented Feb 5, 2015