While tlsfuzzer is currently more of a framework than a full-featured tool,
it still is possible to run tests against common server configurations with a
little bit of effort.
To run the scripts you will need 3 libraries:
It's common that six is already installed, or is available from the operating
system repository.
Alternatively it can be installed by running pip install six as root.
The other two can be installed (again, using pip install ecdsa and
pip install tlslite-ng) or they all can be downloaded to a single location to
minimise dependence on root privileges and permanent changes to the system. For
the rest of this tutorial, I'll follow the latter approach.
Note: the above libraries and tlsfuzzer support both Python 2 and Python
3, but they require at least Python 2.6. If you are running a modern
distribution (RHEL 6 or later), using the provided python in the below
commands is sufficient, if you're running older distribution, you will need to
install new python, and use it for the below commands, usually switching
python to python26 (or similar).
In other words, if you have six already installed, the environment can be
prepared by running the following commands:
git clone https://github.com/tlsfuzzer/tlsfuzzer.git
cd tlsfuzzer
git clone https://github.com/warner/python-ecdsa .python-ecdsa
ln -s .python-ecdsa/src/ecdsa/ ecdsa
git clone https://github.com/tlsfuzzer/tlslite-ng .tlslite-ng
ln -s .tlslite-ng/tlslite/ tlslite
When all dependencies are downloaded or installed, place yourself in the root
directory of the project (one with tlsfuzzer, tests and scripts
directories) and you can start running tests.
All tests support a minimum set of parameters:
-hto specify the hostname of the server under test (tests usually default tolocalhost)-pto specify the port of the server under tests (tests default to 4433)--helpto display all options supported by a given script- names of the scenarios that are to be run (if not provided, all tests in a script are run)
so to test if a server running on example.com on port 433 is not vulnerable
to the
Bleichenbacher attack,
you need to run the following command:
PYTHONPATH=. python scripts/test-bleichenbacher-workaround.py -h example.com -p 443
A run of it will look something like this:
(beginning omitted)
zero byte in last byte of random padding ...
OK
zero byte in first byte of random padding ...
OK
Test end
successful: 8
failed: 0
That prints the names of tests being run (e.g. "zero byte in first byte of random padding") and the overall test result, that 8 tests were run and the server behaved as expected, and in 0 situations the test failed.
Note: unless stated otherwise in the help message of a specific test case, the scripts usually expect a HTTP server with no client certificate authentication.
In such case (where there were no errors), that usually ends the testing and identifies the server as following the relevant RFC documents (like RFC 5246) or not vulnerable to the vulnerability.
To be able to read the error messages of tlsfuzzer scripts, it's necessary to
know a little about how it works internally.
The simplest test script is the test-conversation.py, in it you will find a lot of boilerplate, and a single test scenario:
conversation = Connect(host, port)
node = conversation
ciphers = [CipherSuite.TLS_RSA_WITH_AES_128_CBC_SHA,
CipherSuite.TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
node = node.add_child(ClientHelloGenerator(ciphers))
node = node.add_child(ExpectServerHello())
node = node.add_child(ExpectCertificate())
node = node.add_child(ExpectServerHelloDone())
node = node.add_child(ClientKeyExchangeGenerator())
node = node.add_child(ChangeCipherSpecGenerator())
node = node.add_child(FinishedGenerator())
node = node.add_child(ExpectChangeCipherSpec())
node = node.add_child(ExpectFinished())
node = node.add_child(ApplicationDataGenerator(
bytearray(b"GET / HTTP/1.0\n\n")))
node = node.add_child(ExpectApplicationData())
node = node.add_child(AlertGenerator(AlertLevel.warning,
AlertDescription.close_notify))
node = node.add_child(ExpectAlert())
node.next_sibling = ExpectClose()
conversations["sanity"] = conversation
What this code does, is it sets up a list of things to do and things to expect from the server. In this case, it performs a simple RSA key exchange, an HTTP GET request and expects data back.
Nodes with Generator in the name are creating messages and sending them to
the server, nodes that have Expect in the name will pause execution (with a
timeout) and wait for a message from the server.
In some cases, multiple behaviours are acceptable, this is implemented using
the "siblings". In the above example, after sending the close_notify Alert to
server, the test case expects the server to either send an alert of its own
(ExpectAlert), or to just close the connection (ExpectClose).
If a node has no children, an implicit (TCP) connection close is placed there.
Expect nodes not only verify that the messages received match the expected type, but also that it is matching other messages (e.g. that the Server Hello doesn't advertise support for extensions that the Client Hello didn't include). In other words, unless there are specific options set, tlsfuzzer will behave as a strict, well-behaved TLS client.
Scripts expect specific behaviour from a server, not specific error or message.
In other words, a passing test is a test to which the behaviour of the server matched the expectation (in some cases that may mean that server did report a failure through Alert message). A failing test is a test in which the server did not match expected behaviour - for example did not detect an error it should have detected and continued handshake.
Typical error message looks like this:
zero byte in first byte of random padding ...
Error encountered while processing node <tlsfuzzer.expect.ExpectAlert object at 0x7f96e7e56a90> (child: <tlsfuzzer.expect.ExpectClose object at 0x7f96e7e56ad0>) with last message being: <tlslite.messages.Message object at 0x7f96e79f4090>
Error while processing
Traceback (most recent call last):
File "scripts/test-bleichenbacher-workaround.py", line 250, in main
runner.run()
File "/root/tlsfuzzer/tlsfuzzer/runner.py", line 178, in run
node.process(self.state, msg)
File "/root/tlsfuzzer/tlsfuzzer/expect.py", line 571, in process
raise AssertionError(problem_desc)
AssertionError: Expected alert description "bad_record_mac" does not match received "handshake_failure"
The first line, informs us which scenario was running when the error occurred, in this case "zero byte in first byte of random padding".
Second line tells us which node was being processed, in this case it was
ExpectAlert and its child is ExpectClose.
Finally, the last line tells us what went wrong, in this case the error was
that the test expected an alert with bad_record_mac but got
handshake_failure.
Pattern:
Error encountered while processing node <tlsfuzzer.expect.ExpectAlert ...
...
AssertionError: Expected alert description "bad_record_mac" does not match received "handshake_failure"
Situations where the received alert does not match the expected one may be
sign of a vulnerability (like in the example with Bleichenbacher test above),
bug in implementation under test or RFC non-compliance. In rare cases it may
be caused by test case being too strict (e.g. in some situations sending
insufficient_security alert when the test expects handshake_failure is
valid behaviour).
Pattern:
`Error encountered while processing node <tlsfuzzer.expect.ExpectServerHello ...
...
AssertionError: Unexpected message from peer: Alert(fatal, handshake_failure)``
handshake_failure alert received in place of Server Hello usually means that
the server did not accept any cipher or any extension settings we sent to it.
This may happen when the server has only an ECDSA certificate (support for them
is not advertised in many scripts in in tlsfuzzer)
or did not enable ciphers which are necessary for the test being run.
Most tests require TLS_RSA_WITH_AES_128_CBC_SHA cipher to be enabled. In cases where the test checks handling of messages not applicable in RSA key exchange, the ciphers used are other variants of the AES ciphersuites. Inspect specific script to know more or read its help message.
Pattern:
Error encountered while processing node <tlsfuzzer.expect.ExpectServerHelloDone
...
AssertionError: Unexpected message from peer: Handshake(certificate_request)
Situation like this means that the server asked the client for certificate (did send Certificate Request message) but the client (tlsfuzzer) did not expect that. In cases like this either the server should be reconfigured to not ask for client authentication, or a different test script should be used.
Pattern:
Error encountered while processing node <tlsfuzzer.expect.ExpectAlert ...
...
AssertionError: Unexpected message from peer: ApplicationData(len=8000)
Note: for most tests it will be ExpectAlert, but in general, we're
looking for a node right after ExpectApplicationData.
Test cases in general expect just one TLS record as a response to the HTTP GET query. That limits the response to 16384 bytes (16kiB).
Situations where it is OK to send more than one Application Data message:
- when the response is larger than 16KiB
- when the server is optimised for latency (Time to first byte) and all messages but the last have the same size (e.g. 4KiB, 4KiB, 4KiB and 277B would be OK)
- when the negotiated protocol is TLS 1.0 and the negotiated cipher suite is using CBC mode, then the first message can be 1 or 0 bytes long, and others just as above (this is mitigation of the BEAST)
- the server under test is an echo server, not HTTP, then it may send two Application Data packets as it will receive two lines as the input
Unfortunately that complexity means that the analysis needs to be performed manually, using a tool like Wireshark.
Some situations where multiple Application Data messages being sent is not ok:
- splits happening on line end - that leaks the line lengths to a passive observer
- splits happening on HTTP headers end - that leaks the size of headers to a passive observer
- 1/n-1 split in TLS 1.1 or later or in stream ciphers - it's unnecessary and wastes bandwidth
Pattern:
Error encountered while processing node <tlsfuzzer.expect.ExpectServerHello ...
...
AssertionError: Required extension renegotiation_info missing
Note: the specific extension depends on test case.
Error of this kind usually means that the server does not support functionality
necessary for the test or, in case of renegotiation_info does not support a
feature that many other servers consider mandatory.
Pattern:
Error encountered while processing node <tlsfuzzer.expect.ExpectCertificate ...
...
File "/home/hkario/dev/tlsfuzzer/tlslite/messages.py", line 1128, in _parse_tls12
x509.parseBinary(certBytes)
File "/home/hkario/dev/tlsfuzzer/tlslite/x509.py", line 92, in parseBinary
raise SyntaxError("Unrecognized AlgorithmIdentifier")
SyntaxError: Unrecognized AlgorithmIdentifier
This is an indication that the server has sent a certificate with an DSA or
EdDSA key. They are currently unsupported in tlsfuzzer or tlslite-ng.
Reconfigure the server to use RSA or ECDSA certificates.
If a server already is configured with ECDSA or RSA certificates, it indicates that the system for selecting correct certificate (or certificate chain) when the client does not adverise support for given key type is not working correctly.
Pattern:
Error encountered while processing node <tlsfuzzer.messages.Connect ...
...
sock.connect((self.hostname, self.port))
File "/usr/lib64/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
and
Error encountered while processing node <tlsfuzzer.messages.Connect...
...
File "/usr/lib64/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
timeout: timed out
The hostname or the port are incorrect for the system or a firewall on the way blocks communication.
In other words: communication failed before TLS got involved.
Pattern:
Error encountered while processing node <tlsfuzzer.expect.ExpectServerHello ...
...
AssertionError: Unexpected closure from peer
Note: it may happen at any node, though most commonly on
ExpectServerHello and ExpectAlert.
Some TLS implementations (or some combinations of TLS implementation and an
application) do not send alerts. This makes testing such implementations much
harder, and verifying that the exchanged messages do not cause unintended
behaviour in the server requires running the tests with valgrind, ubsan, asan
and extended logging on server side. This makes running the tests and verifying
results much harder and specific for that one implementation. Because
tlsfuzzer aims to be a universal test suite and RFC conformance checker,
test cases are not written in a way that allows the server to not send alert
messages.
Case in point: if test-bleichenbacher-workaround.py would be written in a way
that server can respond either with an alert or by closing the connection, a
vulnerable behaviour in which the server sometimes sends the correct alert and
sometimes closes the connection would not be detected, reporting a false
negative to the user.
This test requires a HTTP server with a RSA ciphersuite (TLS_RSA_WITH_*_*)
enabled and not asking for client certificates (only for GnuTLS it's not a
default configuration).
In case the server is well-behaved (responds with handshake_failure in case
it cannot negotiate the client proposed ciphersuite) and does not support any
RSA ciphersuite tlslite-ng supports, only the "sanity" tests will fail.
This does not mean that the implementation is not vulnerable, only that the
given configuration isn't (assuming that the server does not implement any
uncommon ciphers, like Camellia, Aria or others unsupported by tlslite-ng).
A good test requires a server configuration that enables all RSA ciphers that a given implementation supports (or, if implementation does not allow for enabling some groups of ciphers together, multiple runs that together had all ciphers enabled).
The test is tuned for testing over a WAN link. If the server is on a local
network, it is possible to speed up test execution significantly by passing the
option -t 0.01 (in general, that number should be twice as big as the RTT to
the server, in seconds). Setting it too low can cause the test case to report
false
negatives!
While the test allows for setting the expected alert response to a Finished
message sent after malformed Client Key Exchange (using the -a option) the
alert sent must be the same for all tests (that is, if one half of scenarios
pass with default configuration and other pass with -a 0 option set, it makes
the server vulnerable to the Bleichenbacher attack). In general, the
workaround requires the server not to treat the Finished message specially,
so the alert sent should be the same as the one generated while running
test-fuzzed-MAC.py.
Also note that if setting this option is necessary, it shows that the server is
not RFC compliant, which in turn, as you can see, makes testing it harder and
more complex.
While not testing for Bleichenbacher directly, the tests test-invalid-rsa-key-exchange-messages.py and test-truncating-of-kRSA-client-key-exchange.py perform checks related to RSA key exchange. Failures there may be a sign of other problems.
This test is very stict on the size of returned reply and expects the reply to be sent in a single record, if its size allows for that.
To quickly check what is the size that the server sends, it's possible to run the test with this options:
test-record-size-limit.py --reply-AD-size 1 \
'check if server accepts maximum size in TLS 1.2'
or:
test-record-size-limit.py --reply-AD-size 1 \
'check if server accepts maximum size in TLS 1.3'
It should fail with
AssertionError: ApplicationData of unexpected size: X, expected: 1
where X is the value that needs to be passed as argument to --reply-AD-size.
To make the test case more deterministic, it is possible to specify the
exact (HTTP) request being sent to server using the --request option. For
example, to perform a GET request for a specific file, not the / object.
--cookie option must be used if the server sends cookie extension in
HelloRetryRequest message.
--supported-groups must to be used when the server sends supported_groups
in EncryptedExtension during a normal TLS 1.3 handshakes while
--hrr-supported-groups must be used when the server sends that extension
in handshakes that force the server to send HelloRetryRequest message.
In case the server limits the size it advertises in its extension,
then --expect-size can be used to set the expected limit. Note though,
the value expected from server in TLS 1.3 will be one byte larger than that for
TLS 1.2 and earlier protocols as this is the expected behaviour of servers that
support the biggest possible records. It also makes the size of actual
application data payload the same irrespective of negotiated protocol version.
This test verifies server's support for different Signature Algorithms for client certificates and CertificateVerify messages. It tests all the algorithms supported by tlsfuzzer, and verifies that only the advertised ones are accepted by the server. It also verifies that algorithms incompatible with the certificate type provided (e.g rsa_pss_rsae_sha256 with "rsa-pss" certificates) are refused as well. It also fuzzes signatures in various ways to make sure servers behave according to TLS1.3 specifications.
NOTE: To test all algorithms it is necessary to run this test twice, once with an "rsa" certificate and once with an "rsa-pss" certificate.
The list of server supported Signature Algorithm's must be provided via the -s
command line option, the default is set to match tlslite-ng sigalg selection.
The algorithms can be defined both via shorthand strings, type+hash strings,
type and hash can also be expressed via numeric ids.
Example: -s "sha256+rsa rsa_pss_pss_sha384 8+11"
For some correctness tests, only one among multiple algorithms of the same type will be used. For example there is a test that checks that a "correct" signature using one hash but being advertised as using different hash is refused. The test machinery will select a signature algorithm matching the first hash in an ordered list and use the second hash in the actual signature. The ordered list of hashes to select from can be changed using the -o parameter. The default is "sha256 sha384 sha512", in that order. The test that uses the wrong mgf1 hash, assuming a server that accepts all standard rsa algorithms, will choose an algorithm that uses the sha256 (the first in the list) for the envelope, and actually uses sha384 (the second in the list) for the signature operation. To test different combinations one can simply change the order to something like -o "sha512 sha256 sha384"; this will cause the test to send an envelope for rsa_pss_pss_sha512 (for "rsa-pss" certificates), but will use sha256 as the actual mgf1 hash when generating the signature. Other tests also uses the ordered hash list to choose an algorithm among multiple viable ones advertised by the server and can similarly be affected by changing the ordered list of hashes. At least two hashes must be provided in the list, although only one need to be supported by the server. If a hash is not supported by the server it will be skipped in test that select algorithms that need to be supported by the server, skipped hashes may still be selected to generate signatures or invalid envelope values as needed by the test.