JSON formatter shows incorrect line number for steps in a Scenario Outline

[sjieg]

Copied issue from cucumber-ruby: cucumber/cucumber-ruby#1108

Configuration

• Cucumber: 2.* (Confirmed issue on 2.4.0 & 2.0.0)
• Gherkin: 4.1.1

Issue not present on Cucumber 1.3.20, that's why I assume it came with version 2.
Upon further inspection, it seems the introduction of Cucumber Core caused this issue

Summary

The line numbers given to a step in the json formatter is always the line number of the Example row. Because of this, it's not possible to distinguish the steps from each other, as the line number is their unique value.

Expected Behavior

In Cucumber 1.3.20 the correct JSON output is given, I've cut out the important pieces:

Feature: This is line 1

  Scenario Outline: This is line 3
    Given this step is line 4
    When when this step uses "<outline-item>" this should be line 5
    Then this is line 6
    Examples:
      | outline-item   |
      | example-line-9 |

cucumber --f json >> output.json

[
  {
    "keyword": "Feature",
    "name": "This is line 1",
    "line": 1,
    "elements": [
      {
        "keyword": "Scenario Outline",
        "name": "This is line 3",
        "line": 9,
        "steps": [
          {
            "keyword": "Given ",
            "name": "this step is line 4",
            "line": 4 # <---
          },
          {
            "keyword": "When ",
            "name": "when this step uses \"example-line-9\" this should be line 5",
            "line": 5 # <---
          },
          {
            "keyword": "Then ",
            "name": "this is line 6",
            "line": 6 # <---
          }
        ]
      }
    ]
  }
]

Note how the step line numbers are the same as the actual line number of the step.

Current Behavior

In cucumber 2.4.0 the following JSON output is given:

[
  {
    "uri": "features/outline_test.feature",
    "id": "this-is-line-1",
    "keyword": "Feature",
    "name": "This is line 1",
    "description": "",
    "line": 1,
    "elements": [
      {
        "id": "this-is-line-1;this-is-line-3;;2",
        "keyword": "Scenario Outline",
        "name": "This is line 3",
        "description": "",
        "line": 9, # <--- This is correct, it refers to the line number of the example
        "type": "scenario",
        "steps": [
          {
            "keyword": "Given ",
            "name": "this step is line 4",
            "line": 9 # <--- Should be line 4
          },
          {
            "keyword": "When ",
            "name": "when this step uses \"example-line-9\" this should be line 5",
            "line": 9 # <--- Should be line 5
          },
          {
            "keyword": "Then ",
            "name": "this is line 6",
            "line": 9 # <--- Should be line 6
          }
        ],
      }
    ]
  }
]

Context & Motivation

This issue has affected us at spriteCloud in the following way:

• We run Cucumber and screenshots are saved on the server it ran on.
• Then the screenshots are exported to the results database
• Before exporting, the file is linked to the step where the screenshot was taken
  This is going fine with regular Scenario's, but when it is an Outline we're not able to link the screenshot to the step and thus we can't visualize the given error.

Your Environment

• Version used: Cucumber 2.4.0
• Operating System and version: Windows 10, Ubuntu 16 and other linux systems.
• Link to your project: http://calliope.pro

[sjieg]

I've already found the issue and created a fix locally. I'll commit a pull request shortly.

[brasmusson]

I don't think this issue belongs here, since the JSON formatter is implemented in the Cucumber-Ruby project and its spec is also there - therefore issues and discussions about its behaviour should be kept there. Some issues might be fixed by a change here, but issues here in Cucumber-Ruby-Core should use the concepts defined here, rather than for instance formatters of Cucumber-Ruby.

Yes, the behaviour was changed in Cucmber-Ruby v2.0.0 (I do not know how conscious the change was), but both behaviours is equally right and equally wrong. The step executed is not the step of the scenario outline, but an instantiated/expanded step based on the step of the scenario outline - instantiated/expanded with the use of the data of the example row. The newer version of Gherkin get it really right by using multiple locations for compiled steps from scenario outlines (both the line of the scenario outline step and the example row are associated with the compiled step). Similarly in Cucumber-Ruvy >v2.0.0 both the scenario outline step and the instantiate/expanded step are included in error backtraces example:

  features/step_definitions/steps.rb:4:in `/^this step fails$/'
  features/outline.feature:9:in `Given this step fails'
  features/outline.feature:4:in `Given this step <status>'

As using one line is not really correct, regardless of which of the line of the scenario outline step or the
line of the example row is chosen, I undecided about what I think the behaviour should be.

[sjieg]

Thanks for your input @brasmusson please feel free to move the ticket to Cucumber-ruby, I'm not familiar with your processes.

I understand that this is a difficult matter as in the depth that the issue is in and how it could break existing cucumber projects, however, I do not feel it's equally wrong or right. From my point of view, the data that we're working with is a crude form of a database, and thus it should apply to the standards of a database. The most important rule in databases is that in every Row of data should have a unique ID. In our case, the Line number + File creates this uniqueness. With Steps not having a Line number, it isn't unique and so it can't be used as data.

I understand if this change won't be added to Cucumber 2.x, but I hope you will consider implementing it in the upcoming Cucumber 3.

[brasmusson]

the Line number + File creates this uniqueness

3  Scenario Outline:
4    Given <x>
 ...
9    Examples
10  |      x        |
11  | one thing     |
12  | another thing |

Using <path/file.feature>:4 as id for both Given one thing and Given another thing is misleading at best, they are very different steps that could well be match to different step definitions. <path/file.feature>:4,11 for Given one thing and <path/file.feature>:4,12 for Given another thing on the other hand gives the complete picture.

[sjieg]

I agree @brasmusson, if steps would get an id attribute like a feature and feature-elements, then the value should contain both the step and the example-row line number.

But in the case of the line attribute, I keep to my opinion that it should contain only the line number of the step location.

Anyway, what do you ought the chance of this being implemented?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[sjieg]

Haven't had time to look into the discussed solution. But it's still on the to-do list. Feel free to assign to me.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[brasmusson]

I close this because the Json Formatter is not part of the this project, so issues with the Json Formatter should not be raised here. Issues raised here should be formulated in concepts available in this project. This issue with the Json Formatter is tracked in cucumber/cucumber-ruby#1108.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.