Failure results make it impossible to figure out what diverged from the expected string

[kurko]

Summary

Aruba/Cucumber output is cryptic. It's basically impossible to find out what is different between the expected string and the output string when there's a test failure.

Expected Behavior

I would expect to have something like RSpec provides, like:

expected: "string1"
got:      "string2"

or, because my output has so many \n, make a diff between all of them by splitting lines:

expected: "string1\nstring again"
got:      "string2\nstring again"

-string1\n"
+string2\n"

Current Behavior

The problem is seen in this Pull Request: https://github.com/zipmark/rspec_api_documentation/pull/313/files#diff-0b3146a61839789912f87a2e19beaaa4R244. The library is responsible for generating documentation files. We write an expectation and then make sure the file matches what was generated.

However, when there's one character off, this is the output I get:

expected "FORMAT: A1\n\n# Group Instructions\n\nInstructions help the users use the app.\n\n## Instructions Collection [/instructions]\n\n### Returns all instructions [GET]\n\n+ Request List all instructions\n\n    + Headers\n\n        Host: example.org\n\n    + Body\n\n        Content-Type: text/html;charset=utf-8\n        Content-Length: 57\n\n+ Response 200 (text/html;charset=utf-8)\n\n    + Headers\n\n        Content-Type: text/html;charset=utf-8\n        Content-Length: 57\n\n    + Body\n\n        {\"data\":{\"id\":\"1\",\"type\":\"instructions\",\"attributes\":{}}}\n\n# Group Orders\n\nOrders resource\n\n## Orders Collection [/orders]\n\n### Creates an order [POST]\n\n+ Request Creating an order (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 73\n\n+ Response 201 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 73\n\n    + Body\n\n        {\n          \"order\": {\n            \"name\": \"Order 1\",\n            \"amount\": 100.0,\n            \"description\": \"A great order\"\n          }\n        }\n\n### Return all orders [GET]\n\n+ Request Getting a list of orders\n\n    + Headers\n\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 137\n\n+ Response 200 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 137\n\n    + Body\n\n        {\n          \"page\": 1,\n          \"orders\": [\n            {\n              \"name\": \"Order 1\",\n              \"amount\": 9.99,\n              \"description\": null\n            },\n            {\n              \"name\": \"Order 2\",\n              \"amount\": 100.0,\n              \"description\": \"A great order\"\n            }\n          ]\n        }\n\n## Single Order [/orders/{id}]\n\n+ Parameters\n  + id: 1 (required, string) - Order id\n\n+ Attributes (object)\n  + name: a name (required) - The order name\n  + amount - The order amount\n  + description: a description (string) - The order description\n\n### Deletes a specific order [DELETE]\n\n+ Request Deleting an order (application/x-www-form-urlencoded)\n\n    + Headers\n\n        Host: example.org\n        Content-Type: application/x-www-form-urlencoded\n\n    + Body\n\n        Content-Type: text/html;charset=utf-8\n        Content-Length: 0\n\n+ Response 200 (text/html;charset=utf-8)\n\n    + Headers\n\n        Content-Type: text/html;charset=utf-8\n        Content-Length: 0\n\n### Returns a single order [GET]\n\n+ Request Getting a specific order\n\n    + Headers\n\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 73\n\n+ Response 200 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 73\n\n    + Body\n\n        {\n          \"order\": {\n            \"name\": \"Order 1\",\n            \"amount\": 100.0,\n            \"description\": \"A great order\"\n          }\n        }\n\n### Updates a single order [PUT]\n\n+ Request Invalid request (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 0\n\n+ Response 400 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 0\n\n+ Request Update an order (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 111\n\n+ Response 200 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 111\n\n    + Body\n\n        {\n          \"data\": {\n            \"id\": \"1\",\n            \"type\": \"order\",\n            \"attributes\": {\n              \"name\": \"Order 1\",\n              \"amount\": 100.0,\n              \"description\": \"A description\"\n            }\n          }\n        }" to have file content: "FORMAT: A1\n\n# Group Instructions\n\nInstructions help the users use the app.\n\n## Instructions Collection [/instructions]\n\n### Returns all instructions [GET]\n\n+ Request List all instructions\n\n    + Headers\n\n        Host: example.org\n\n    + Body\n\n        Content-Type: text/html;charset=utf-8\n        Content-Length: 57\n\n+ Response 200 (text/html;charset=utf-8)\n\n    + Headers\n\n        Content-Type: text/html;charset=utf-8\n        Content-Length: 57\n\n    + Body\n\n        {\"data\":{\"id\":\"1\",\"type\":\"instructions\",\"attributes\":{}}}\n\n# Group Orders\n\nOrders resource\n\n## Orders Collection [/orders]\n\n### Creates an order [POST]\n\n+ Request Creating an order (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 73\n\n+ Response 201 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 73\n\n    + Body\n\n        {\n          \"order\": {\n            \"name\": \"Order 1\",\n            \"amount\": 100.0,\n            \"description\": \"A great order\"\n          }\n        }\n\n### Return all orders [GET]\n\n+ Request Getting a list of orders\n\n    + Headers\n\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 137\n\n+ Response 200 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 137\n\n    + Body\n\n        {\n          \"page\": 1,\n          \"orders\": [\n            {\n              \"name\": \"Order 1\",\n              \"amount\": 9.99,\n              \"description\": null\n            },\n            {\n              \"name\": \"Order 2\",\n              \"amount\": 100.0,\n              \"description\": \"A great order\"\n            }\n          ]\n        }\n\n## Single Order [/orders/{id}]\n\n+ Parameters\n  + id: (required, string) - Order id\n\n+ Attributes (object)\n  + name: a name (required) - The order name\n  + amount - The order amount\n  + description: a description (string) - The order description\n\n### Deletes a specific order [DELETE]\n\n+ Request Deleting an order (application/x-www-form-urlencoded)\n\n    + Headers\n\n        Host: example.org\n        Content-Type: application/x-www-form-urlencoded\n\n    + Body\n\n        Content-Type: text/html;charset=utf-8\n        Content-Length: 0\n\n+ Response 200 (text/html;charset=utf-8)\n\n    + Headers\n\n        Content-Type: text/html;charset=utf-8\n        Content-Length: 0\n\n### Returns a single order [GET]\n\n+ Request Getting a specific order\n\n    + Headers\n\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 73\n\n+ Response 200 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 73\n\n    + Body\n\n        {\n          \"order\": {\n            \"name\": \"Order 1\",\n            \"amount\": 100.0,\n            \"description\": \"A great order\"\n          }\n        }\n\n### Updates a single order [PUT]\n\n+ Request Invalid request (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 0\n\n+ Response 400 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 0\n\n+ Request Update an order (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Host: example.org\n\n    + Body\n\n        Content-Type: application/json\n        Content-Length: 111\n\n+ Response 200 (application/json)\n\n    + Headers\n\n        Content-Type: application/json\n        Content-Length: 111\n\n    + Body\n\n        {\n          \"data\": {\n            \"id\": \"1\",\n            \"type\": \"order\",\n            \"attributes\": {\n              \"name\": \"Order 1\",\n              \"amount\": 100.0,\n              \"description\": \"A description\"\n            }\n          }\n        }" (RSpec::Expectations::ExpectationNotMetError)
      features/api_blueprint_documentation.feature:247:in `Then the file "doc/api/index.apib" should contain exactly:'

Here's an actual print screen:

Possible Solution

I tried to dig into Aruba's source code but I'm not very familiar with how even Cucumber works. Its documentation is somewhat cryptic to me (described as steps). I see that there's something called @announce, but that didn't do anything for me (and I failed to find documentation that I could use).

RSpec has that functionality to compare strings built-in. Would it be possible to should pipe that output into RSpec's matches?

I could try to help with that but I'd need directions because the documentation is unclear to me.

Steps to Reproduce (for bugs)

1. git clone https://github.com/zipmark/rspec_api_documentation.git
2. git checkout 235-blueprint-api
3. bundle install
4. Make a one-character change to features/api_blueprint_documentation.feature, around lines 260.
5. bundle exec cucumber features

You will see my screenshot, basically.

Context & Motivation

I'm working on a lib that generates documentation automatically based on RSpec tests. The fixture has over 200 lines, so when even one character changes, it's virtually impossible to find out why it's failing, or in other words, what string is different from the expectation.

This is making my work very slow. I have to either look by eye or copy paste somewhere else to figure out the diff.

Your Environment

• Version used: 0.13.0 (perhaps it was fixed in 0.14.x?)
• Operating System and version: OSX, El Capitán
• Link to your project: https://github.com/zipmark/rspec_api_documentation, more specifically (https://github.com/zipmark/rspec_api_documentation/pull/313/files#diff-0b3146a61839789912f87a2e19beaaa4R244)

Thanks a lot.

[mattwynne]

I've experienced the same frustration using aruba in Cucumber-Ruby's own test suite.

The solution, for me, is to use the word 'exactly' in your step, i.e. Then the output should contain exactly). This invokes the exact string matcher meaning you'll get a better diff. Otherwise, it uses a string including matcher, which can't give you as good a diff since it compares using a Regexp.

[maxmeyer]

@kurko I tried to make output more reable by using the diffable keyword in RSpec matchers:

aruba/lib/aruba/matchers/string/output_string_eq.rb

Line 28 in b23f6da

diffable

. If there are more matchers where this does make sense, feel free to provide a pull request.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[stale]

This issue has been automatically closed because of inactivity. You can support the Cucumber core team on opencollective.

[mvz]

This has been partly handled in #546. For negated matchers, the problem persists.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[xtrasimplicity]

Bump.