Handle invalid characters in "Authorization: token ..." headers #3006

osantana · 2015-06-03T17:58:49Z

Resolve #2928.

Request headers are passed as bytes() on Py3 (or str() on Py2), so, we try to decode it before calling self.authenticate_credentials().

If we got an error decoding token we raise an AuthenticationFailed() exception.

xordoquy · 2015-06-03T20:23:01Z

I'll run the test without the fix to make sure there is an issue

osantana · 2015-06-04T00:12:22Z

$ tox -e py27-django18 -e py34-django18
GLOB sdist-make: /home/osantana/Work/django-rest-framework/django-rest-framework/setup.py
py27-django18 inst-nodeps: /home/osantana/Work/django-rest-framework/django-rest-framework/.tox/dist/djangorestframework-3.1.2.zip
py27-django18 installed: Django==1.8,django-filter==0.9.2,django-guardian==1.2.5,djangorestframework==3.1.2,Markdown==2.5.2,py==1.4.28,pytest==2.6.4,pytest-django==2.8.0,six==1.9.0,wheel==0.24.0
py27-django18 runtests: PYTHONHASHSEED='3803568486'
py27-django18 runtests: commands[0] | ./runtests.py --fast
............F.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
================================================================== FAILURES ===================================================================
________________________________________ TokenAuthTests.test_fail_post_form_passing_invalid_token_auth ________________________________________
tests/test_authentication.py:170: in test_fail_post_form_passing_invalid_token_auth
    response = self.csrf_client.post('/token/', {'example': 'example'}, HTTP_AUTHORIZATION=auth)
rest_framework/test.py:168: in post
    path, data=data, format=format, content_type=content_type, **extra)
rest_framework/test.py:90: in post
    return self.generic('POST', path, data, content_type, **extra)
rest_framework/compat.py:231: in generic
    return self.request(**r)
rest_framework/test.py:157: in request
    return super(APIClient, self).request(**kwargs)
rest_framework/test.py:109: in request
    request = super(APIRequestFactory, self).request(**kwargs)
.tox/py27-django18/local/lib/python2.7/site-packages/django/test/client.py:448: in request
    response = self.handler(environ)
.tox/py27-django18/local/lib/python2.7/site-packages/django/test/client.py:122: in __call__
    response = self.get_response(request)
rest_framework/test.py:129: in get_response
    return super(ForceAuthClientHandler, self).get_response(request)
.tox/py27-django18/local/lib/python2.7/site-packages/django/core/handlers/base.py:218: in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
.tox/py27-django18/local/lib/python2.7/site-packages/django/core/handlers/base.py:132: in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
.tox/py27-django18/local/lib/python2.7/site-packages/django/views/decorators/csrf.py:58: in wrapped_view
    return view_func(*args, **kwargs)
.tox/py27-django18/local/lib/python2.7/site-packages/django/views/generic/base.py:71: in view
    return self.dispatch(request, *args, **kwargs)
rest_framework/views.py:456: in dispatch
    response = self.handle_exception(exc)
rest_framework/views.py:444: in dispatch
    self.initial(request, *args, **kwargs)
rest_framework/views.py:358: in initial
    self.perform_authentication(request)
rest_framework/views.py:296: in perform_authentication
    request.user
rest_framework/request.py:489: in __getattribute__
    return super(Request, self).__getattribute__(attr)
rest_framework/request.py:264: in user
    self._authenticate()
rest_framework/request.py:452: in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
rest_framework/authentication.py:173: in authenticate
    return self.authenticate_credentials(auth[1])
rest_framework/authentication.py:177: in authenticate_credentials
    token = self.model.objects.select_related('user').get(key=key)
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/models/query.py:328: in get
    num = len(clone)
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/models/query.py:144: in __len__
    self._fetch_all()
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/models/query.py:965: in _fetch_all
    self._result_cache = list(self.iterator())
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/models/query.py:238: in iterator
    results = compiler.execute_sql()
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py:829: in execute_sql
    cursor.execute(sql, params)
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/backends/utils.py:64: in execute
    return self.cursor.execute(sql, params)
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/utils.py:97: in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/backends/utils.py:64: in execute
    return self.cursor.execute(sql, params)
.tox/py27-django18/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py:318: in execute
    return Database.Cursor.execute(self, query, params)
E   ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.
1 failed, 669 passed, 1 warnings in 5.78 seconds
ERROR: InvocationError: '/home/osantana/Work/django-rest-framework/django-rest-framework/runtests.py --fast'
py34-django18 inst-nodeps: /home/osantana/Work/django-rest-framework/django-rest-framework/.tox/dist/djangorestframework-3.1.2.zip
py34-django18 installed: Django==1.8,django-filter==0.9.2,django-guardian==1.2.5,djangorestframework==3.1.2,Markdown==2.5.2,py==1.4.28,pytest==2.6.4,pytest-django==2.8.0,six==1.9.0,wheel==0.24.0
py34-django18 runtests: PYTHONHASHSEED='3803568486'
py34-django18 runtests: commands[0] | ./runtests.py --fast
............F.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
================================================================== FAILURES ===================================================================
________________________________________ TokenAuthTests.test_fail_post_form_passing_invalid_token_auth ________________________________________
tests/test_authentication.py:170: in test_fail_post_form_passing_invalid_token_auth
    response = self.csrf_client.post('/token/', {'example': 'example'}, HTTP_AUTHORIZATION=auth)
rest_framework/test.py:168: in post
    path, data=data, format=format, content_type=content_type, **extra)
rest_framework/test.py:90: in post
    return self.generic('POST', path, data, content_type, **extra)
rest_framework/compat.py:231: in generic
    return self.request(**r)
rest_framework/test.py:157: in request
    return super(APIClient, self).request(**kwargs)
rest_framework/test.py:109: in request
    request = super(APIRequestFactory, self).request(**kwargs)
.tox/py34-django18/lib/python3.4/site-packages/django/test/client.py:448: in request
    response = self.handler(environ)
.tox/py34-django18/lib/python3.4/site-packages/django/test/client.py:122: in __call__
    response = self.get_response(request)
rest_framework/test.py:129: in get_response
    return super(ForceAuthClientHandler, self).get_response(request)
.tox/py34-django18/lib/python3.4/site-packages/django/core/handlers/base.py:218: in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
.tox/py34-django18/lib/python3.4/site-packages/django/core/handlers/base.py:132: in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
.tox/py34-django18/lib/python3.4/site-packages/django/views/decorators/csrf.py:58: in wrapped_view
    return view_func(*args, **kwargs)
.tox/py34-django18/lib/python3.4/site-packages/django/views/generic/base.py:71: in view
    return self.dispatch(request, *args, **kwargs)
rest_framework/views.py:456: in dispatch
    response = self.handle_exception(exc)
rest_framework/views.py:444: in dispatch
    self.initial(request, *args, **kwargs)
rest_framework/views.py:358: in initial
    self.perform_authentication(request)
rest_framework/views.py:296: in perform_authentication
    request.user
rest_framework/request.py:489: in __getattribute__
    return super(Request, self).__getattribute__(attr)
rest_framework/request.py:264: in user
    self._authenticate()
rest_framework/request.py:452: in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
rest_framework/authentication.py:173: in authenticate
    return self.authenticate_credentials(auth[1])
rest_framework/authentication.py:177: in authenticate_credentials
    token = self.model.objects.select_related('user').get(key=key)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/query.py:325: in get
    clone = self.filter(*args, **kwargs)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/query.py:679: in filter
    return self._filter_or_exclude(False, *args, **kwargs)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/query.py:697: in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/sql/query.py:1301: in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/sql/query.py:1328: in _add_q
    current_negated=current_negated, connector=connector, allow_joins=allow_joins)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/sql/query.py:1200: in build_filter
    condition = self.build_lookup(lookups, col, value)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/sql/query.py:1096: in build_lookup
    return final_lookup(lhs, rhs)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/lookups.py:96: in __init__
    self.rhs = self.get_prep_lookup()
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/lookups.py:134: in get_prep_lookup
    return self.lhs.output_field.get_prep_lookup(self.lookup_name, self.rhs)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/fields/__init__.py:727: in get_prep_lookup
    return self.get_prep_value(value)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/fields/__init__.py:1125: in get_prep_value
    return self.to_python(value)
.tox/py34-django18/lib/python3.4/site-packages/django/db/models/fields/__init__.py:1121: in to_python
    return smart_text(value)
.tox/py34-django18/lib/python3.4/site-packages/django/utils/encoding.py:56: in smart_text
    return force_text(s, encoding, strings_only, errors)
.tox/py34-django18/lib/python3.4/site-packages/django/utils/encoding.py:102: in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)
E   django.utils.encoding.DjangoUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb8 in position 8: invalid start byte. You passed in b'abcd1234\xb8' (<class 'bytes'>)
1 failed, 669 passed, 1 warnings in 5.05 seconds
ERROR: InvocationError: '/home/osantana/Work/django-rest-framework/django-rest-framework/runtests.py --fast'
___________________________________________________________________ summary ___________________________________________________________________
ERROR:   py27-django18: commands failed
ERROR:   py34-django18: commands failed

Infos:

$ uname -a
Linux euclides 3.13.0-54-generic #91-Ubuntu SMP Tue May 26 19:15:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=pt_BR.UTF-8
LC_TIME=pt_BR.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=pt_BR.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=pt_BR.UTF-8
LC_NAME=pt_BR.UTF-8
LC_ADDRESS=pt_BR.UTF-8
LC_TELEPHONE=pt_BR.UTF-8
LC_MEASUREMENT=pt_BR.UTF-8
LC_IDENTIFICATION=pt_BR.UTF-8
LC_ALL=

xordoquy · 2015-06-16T20:38:57Z

Confirmed. Thanks for all the work !

Handle invalid characters in "Authorization: token ..." headers

tomchristie · 2015-06-22T13:20:59Z

Nice work! :)

Handle invalid characters in headers

7ae71de

osantana changed the title ~~Handle invalid characters in headers. Fix #2928~~ Handle invalid characters in "Authorization: token ..." headers Jun 3, 2015

osantana mentioned this pull request Jun 3, 2015

Invalid Unicode byte in Authorization token raises an DjangoUnicodeDecodeError #2928

Closed

xordoquy added the Bug label Jun 4, 2015

xordoquy modified the milestones: 3.1.3 Release, 3.1.4 Release Jun 4, 2015

xordoquy added a commit that referenced this pull request Jun 16, 2015

Merge pull request #3006 from osantana/master

139ba7a

Handle invalid characters in "Authorization: token ..." headers

xordoquy merged commit 139ba7a into encode:master Jun 16, 2015

tomchristie modified the milestones: 3.1.4 Release, 3.2.0 Release Jul 30, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle invalid characters in "Authorization: token ..." headers #3006

Handle invalid characters in "Authorization: token ..." headers #3006

osantana commented Jun 3, 2015

xordoquy commented Jun 3, 2015

osantana commented Jun 4, 2015

xordoquy commented Jun 16, 2015

tomchristie commented Jun 22, 2015

Handle invalid characters in "Authorization: token ..." headers #3006

Handle invalid characters in "Authorization: token ..." headers #3006

Conversation

osantana commented Jun 3, 2015

xordoquy commented Jun 3, 2015

osantana commented Jun 4, 2015

xordoquy commented Jun 16, 2015

tomchristie commented Jun 22, 2015