Compiler design and pickles #12
Comments
|
There's already an OSS project called Pickles in the Cucumber/Specflow space. Are you happy overloading that name?
|
|
@sebrose thanks for the heads up about http://www.picklesdoc.com/ We should avoid name overloading. It doesn't matter too much what it's called - it's a lib used by implementors only - it will never be user-visible. I think testicle would be great, but I'm worried some people might make a fuss about it. People have made a fuss about smaller things. It doesn't matter - gherkle, icicle, whatever. I know you probably have great ideas for names, but let's not derail this discussion ;-) |
Understood.
Agreed
|
|
@aslakhellesoy I'm wondering how the formatting layer would deal with scenario outlines if cucumber only deals with PickledCase objects. Same for formatting feature backgrounds. Would it have to deal with both the gherkin AST and the run output to match them by location ? And when would the filtering of features/scenarios based on tags would happen to decide whether they should be run ? Tags don't seem to appear at all in your PickledCase |
|
@stof regarding formatting I'm not sure! The funny thing with Background is that they run several times, so printing them once always seemed a little weird to me. The pickles could have a back-reference to the AST nodes, making it easier to write a formatter that prints the original structure, but that also means we're maintaining the tight coupling to Gherkin I'd like to get away from. Can anyone suggest any smart algorithms/design patterns here? Regarding tags - they will be compiled into the pickles too - I just haven't bothered with it yet since I'm still trying out different designs. |
|
I think the gherkin lib should only deal with parsing text to some For example for XML you have the choice to use a SAX-Parser (emitting I don't think that the 'pickles' belong to 'gherkin'. But maybe I'm missing
|
|
I think the logic, how to interpret the Gherkin language (e.g. background steps are injected before every scenario), scenario outline text placeholders are substituted, tags are "inherited") are belonging the the language semantics. So it is not that bad to keep them together, so that we have a standardized semantics across the tools. The repo is called Visitor: My vote is not to use visitors in the AST, but the different usages (compiler, formatting) should traverse the tree as they want. |
|
@gasparnagy the question is whether the compiler and the parser should live in the same package or be fully separate (with the compiler depending on the parser of course) |
|
I agree with @muhqu that the parser shouldn't do anything else than producing an AST. The parser has no dependencies on the compiler. The compiler depends on (consumes) the AST produced by the parser. The question is whether we put the compiler in the same git repo/library as the parser. I agree with @gasparnagy - I think we should. It could of course live in a separate git repo, but I think there are several benefits in keeping them in the same repo:
I think the pickles (the output of the compiler) should live in a separate library however. So this is what I have in mind: Makes sense? (I love Monodraw) |
|
Makes sense to me @aslakhellesoy. I think Cucumber might also need a dependency on the various compilers, but that's a side issue. On names, I'm comfortable sticking with the name |
|
I've given this some more thought, and I think that while we're all working together to figure out what the structure of those pickles should be, the most pragmatic approach is to keep it all in the Gherkin3 repo/library: Once it stabilises we can re-evaluate whether it makes sense to move pickles and perhaps the compiler too into separate projects. |
|
Another approach is to refactor from the existing Ruby codebase, which already has a pickles in the form of cucumber-ruby-core. We’d just need to make it work with the new AST. |
|
I had a look at the current WIP version. I think it is fine. I have two comments:
|
e.g. https://github.com/cucumber/cucumber-ruby-core/blob/master/spec/cucumber/core/test/case_spec.rb#L100 https://github.com/cucumber/cucumber-ruby-core/blob/master/spec/cucumber/core/test/case_spec.rb#L100 |
|
@mattwynne yep, good point. we do something similar in SpecFlow. There we check if the first column has unique values and if yes, we use that instead of #1, #2, etc. But my question was rather the beginning of the string, so whether it should be prefixed with the keyword ("Scenario: outline name, examples name (#1)" vs "outline name, examples name (#1)"). |
|
Or if there should be a keyword/name separation in the pickle level, as for test cases in cucumber-ruby-core. In Cucumber v2.0 that separation enables that when hooks access the name of the running test case they get the name ( |
|
Keep in mind that we want to generate pickles from other sources than Gherkin. Markdown is high on my list. If we mandate the use of a keyword in pickles I think we're limiting our freedom to use other sources where a keyword isn't natural. |
|
@aslakhellesoy so Pickles is what the README is referring to as Test cases? …if so, then why not just write a simple tool that transforms the AST-JSON to some TestCase-JSON? Sry, if I'm missing the point here. :-/ In code what's on my mind...
Feature: Consuming Cucumbers
Scenario: eat 5 out of 12
Given there are 12 cucumbers
When I eat 5 cucumbers
Then I should have 7 cucumbers
{
"location": { "line": 1, "column": 1 },
"type": "Feature",
"tags": [ ],
"language": "en",
"keyword": "Feature",
"name": "Consuming Cucumbers",
"scenarioDefinitions": [
{
"location": { "line": 3, "column": 3 },
"type": "Scenario",
"keyword": "Scenario",
"name": "eat 5 out of 12",
"steps": [
{
"location": { "line": 4, "column": 5 },
"type": "Step",
"keyword": "Given ",
"text": "there are 12 cucumbers"
},
{
"location": { "line": 5, "column": 6 },
"type": "Step",
"keyword": "When ",
"text": "I eat 5 cucumbers"
},
{
"location": { "line": 6, "column": 6 },
"type": "Step",
"keyword": "Then ",
"text": "I should have 7 cucumbers"
}
],
"tags": [ ]
}
],
"comments": [ ]
}
{
"cases": [
{
"name": "Consuming Cucumbers: eat 5 out of 12",
"sources": [ { "file":"some.feature", "line": 3, "column": 3 } ],
"steps": [
{
"sources": [ { "file":"some.feature", "line": 4, "column": 5 } ],
"text": "there are 12 cucumbers"
},
{
"sources": [ { "file":"some.feature", "line": 5, "column": 6 } ],
"text": "I eat 5 cucumbers"
},
{
"sources": [ { "file":"some.feature", "line": 6, "column": 6 } ],
"text": "I should have 7 cucumbers"
}
]
}
]
} |
Yes. I've updated the README.
That's what Compiler.java does, except that it transforms AST objects to Pickle objects - it doesn't operate on the JSON level. Keep in mind this is work in progress / in flux. We'll have to integrate it into a Cucumber impl to see how well the abstraction works.
Read my updates to the README again and tell me if anything is still unclear. |
|
Yeah I agree that Pickes should not know about Gherkin, very important. The way our stuff currently works on Ruby core, you could take a Pickles Test::Case and ask it to describe its source to you. You’d get callbacks like step, scenario, background etc that tell you where the thing came from, so you could use those to get the keyword (or the bare name) if you needed it. Seems to me that a Pickle’s name should probably include the keyword, though it might be a bit a bit weird with an example row. |
It's fine now. I think I understand now. So, the Pickles object hierarchy represents a materialized view on the steps per test case (scenario). But each step object in these test cases keeps a reference to the original step node (/object) from the AST. The runner (cucumber) only needs to match its step definitions against the steps from the pickles hierarchy to run the test cases. But where will the runner report the results to? e.g. which steps passed, failed, where skipped or ignored because there was no matching step def found... Should the Pickles model (and the Gherkin model) also provide the capability to track the result of a step being run? …If the runner should not need to know about the Pickles source (being it Gherkin, markdown, u-name-it...) and how to render a source specific report, there needs to be some step result propagation between the Pickles model and the source model (Gherkin AST)... I mean, ideally it should be capable of rendering the raw gherkin feature file with some highlighting which steps passed, failed or where skipped. Of course this step state propagation is not |
|
Currently there are two different approaches how to report results from the execution of feature files, one in Cucumber(-Ruby) and one in Cucumber-JVM. Cucumber(-Ruby) follow the feature file closely (background only reported once, normally reporting the result of test cases from Scenario Outlines on the example table row), whereas Cucumber-JVM follow the executed test cases more closely (background reported every time it is executed, report each step executed for test cases from Scenario Outlines). When saying "ideally it should be capable of rendering the raw gherkin feature file with some highlighting" you are asking for what Cucumber(-Ruby) currently does @muhqu , but that is not the only approach as shown by the current behavior of Cucumber-JVM. In case of reports intended for other tools to consume, like currently the json-report produced by the json-formatters (read by masterthoughts cucumber-reporting tools, bamboo-plugins and probably more), it is IMHO essential that there is an explicit result reported for each step executed (for instance one result for each time a background step was executed). |
|
@brasmusson I'm absolutely aware that there are various different report output formats. What I wanted to highlight is that it should be _capable_ of rendering the raw gherkin feature with highlighting. I was just wondering how that could/would be achieved and to which software component this rendering would belong to. Maybe it's a good idea to keep the parsing and rendering of test cases results (in the test cases native source format) in the same place? |
|
I'd like to change the way reporting is done. I don't like the way it's done in either Cucumber-Ruby or Cucumber-JVM. I think it was a mistake to include the source (AST) in the JSON report. It makes everything very complicated. The reporting API can be much simpler. All we need to report is the results, and a link to the input source (file, line and column). Something like this: {
"results": [
{
"uri": "file:///path/to/the.feature",
"location": {"line": 22, "column": 4},
"status": "failed",
"error": "some stack trace",
"startTimeMillis": 1428567433625,
"durationMillis": 12
}
]
}Generating nice looking reports (HTML) can then be done by reading in the source again (which is easy with the new Gherkin3 API), and merge in the results. The API could be something like this (Java): public interface ResultsPlugin {
void testRunStarted();
void testStarted(TestCase testCase);
void testStepStarted(TestStep testStep);
void testStepFinished(TestStep testStep, TestStepResult result);
void testFinished(TestCase testCase);
void testRunFinished();
}Note that we're not using Pickles here, but TestCases. A TestCase wraps a Pickle, and a TestStep wraps a PickleStep (and a Step Definition). A TestStep can also wrap just a before hook or after hook. The difficult bit is the pretty formatter. I'm toying with the idea that when the pretty formatter receives a This approach should work well when scenarios are run sequentially, in a single thread. We're going to make Cucumber able to run scenarios in random order, possibly multi-threaded. In this case I think we should either disable the pretty formatter, or change its behaviour. It doesn't make sense to pretty-print scenarios (especially the feature headers) in random order. |
Reading the source again and having to match the results to Scenario's Steps by location (line/column) doesn't feel that natural, but probably would be a feasible solution.
Ok, but what does the Test(Case|Step) adds to the wrapped Pickle(Case|Step)? …the matched Step Def callback (if any)? |
|
Bear in mind that as of Cucumber 2.0 much of what @aslakhellesoy describes is already live in Cucumber Ruby. For example, the new formatter API, which I admit is currently poorly documented, now works exactly like that. I think this discussion might be informed by some reading of the cucumber-ruby-core source code, notably the runner. And the compiler which works off the Gherin2 AST at the moment. @aslakhellesoy and I have some time in the same room next week so hopefully we can start trying to move this code over to work with Gherkin3 and see what problems come out of it. |
|
@muhqu - yes - it adds the stepdef. There is also a flavour for hooks. This is what I have in mind (Java): interface TestStep {
TestStepResult run();
}
public class PickleTestStep implements TestStep {
public PickleTestStep(Pickle p, StepDefinifion sd) {}
}
public class HookTestStep implements TestStep {
public HookTestStep(HookDefinifion hd) {}
} |
|
@mattwynne I hav to admit that I'm fairly new to 'cucumber' as a processor for gherkin. little bg: I worked like 4 years with behat and just recently switched to cucumber-jvm… with ruby I hav not much to do and like to keep it that way.. I have passion for go and gherkin, that's why I wrote the gherkin-go and go-gherkin parsers ;-) @aslakhellesoy awesome, I think we're now on the same page… 👏 |
|
I'm glad it's becoming more clear. The various cucumber implementations are quite different on the inside, and I'm hopeful that we can document a high-level architecture that will make it easier for developers to make more consistent implementations! The discussions we're having here are very useful. I'll do my best to document the important design decisions about Cucumber over at the cukes.info site. I think the new Gherkin architecture is already documented quite well in this project :-) |
|
@muhqu I was wondering, wouldn't it be better to use an RDF vocab with e.g. JSON-LD by the AST format instead of using plain JSON? I wasn't checked the compiler yet, but JSON-LD transformations might come in handy and the AST would have more semantic either. It could be something like this roughly, but I am not a JSON-LD expert: gherkin Feature: Example feature
As a user of cucumber.js
I want to have documentation on cucumber
So that I can concentrate on building awesome applications
Scenario: Reading documentation
Given I am on the Cucumber.js GitHub repository
When I go to the README file
Then I should see "Usage" as the page titleAST RDF {
"@context": {
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
"cucumber": "http://cucumber.org/vocab.jsonld#"
},
"@id": "http://example.com/features#",
"features": [
{
"@type": "cucumber:Feature",
"label": "Example feature",
"description": [
"As a user of cucumber.js",
"I want to have documentation on cucumber",
"So that I can concentrate on building awesome applications"
],
"scenarios": [
{
"@type": "cucumber:Scenario",
"label": "Reading documentation",
"steps": [
{
"@type": "cucumber:Given",
"label": "I am on the Cucumber.js GitHub repository"
},
{
"@type": "cucumber:When",
"label": "I go to the README file"
},
{
"@type": "cucumber:Then",
"label": "I should see "Usage" as the page title"
}
]
}
]
}
]
} |
|
Currently, the only reason the AST (and compiled AST) have a JSON representation is for shared tests. The shared tests compare the JSON representation of the AST with an expected JSON document. The JSON representation isn't currently used for anything else, but I can imagine it could be used for rendering/reporting - you could take a Gherkin AST as JSON and use it to generate a pretty HTML document for example. @Inf3rno can you give us a concrete example of what capabilities JSON-LD would give us that the current format doesn't give us? It appears the change would only require changing |
|
JSON can describe only hierarchical data structures, while JSON-LD can describe graphs. With this structure it does not matter, but by using tags you'll need to use graph, if you add properties to each tag, not just a label. Another feature that it can annotate the properties with metadata, because it is an RDF format. E.g. the It is hard to tell whether you really need it. I think Markus could tell you more about the advantages and disadvantages of having an RDF vocab, he is an expert in the topic. Probably having an RDF vocab would help to reuse feature descriptions amongst projects. So for example by testing common features like auth (login, logout), user catalog (edit profile, change password, reset password, etc...) or webshop (product catalog, cart, checkout, shipping), etc... It would be nice to import existing feature descriptions, select the ones needed for the project and implement only the step definitions. If you can add another standard vocab, which contains the features, then it would be possible to write automated clients, which can work by any application, which shares e2e tests publicly. Anyways, most of the testing code can be reused to write automated clients, for example by testing a REST API this can be very useful for 3rd party developers. But that's just a vision, probably it is beyond the boundaries of this project. |
Wouldn’t breadth-first also better suit parallel processing? It would be nice to plan for that possible future, especially for larger test suites (or frequent builds on CI servers)… |
|
The compiler isnow released (java, ruby, python, js) so I'm closing this |
Let's use this issue to discuss how to design the Gherkin compiler, which is described at a very high level in the toplevel README. My goal is to decouple Cucumber implementations from Gherkin so Cucumber implementations can consume other formats.
I've been playing around with various designs to implement the compiler - both a visitable AST and a plain old external iteration one. I'm currently leaning towards a non-visitor one because I think a visitable one adds unnecessary complexity.
IMO visitor is only really useful when there is a high degree of polymorphism in the child nodes. In the Gherkin AST it's only
Scenario|ScenarioOutlineandDataTable|DocStringthat are polymorphic.If we stick with visitor we have to decide where to put iteration - in the nodes or in the visitor. For the compiler I think a depth-first traversal is best. For pretty-printing I think a breadth-first traversal is best. In other words, if we implement iteration in the nodes we can't please everyone (unless we introduce a pluggable tree walker).
Furthermore, I'd like all implementations to be as similar as possible. The JavaScript implementation doesn't even define types for the AST (they are just dumb objects with properties). I love the simplicity of that and I'd be sad to have to complicate it.
My current feeling is to not use visitor at all, and let the compiler traverse the AST entirely on its own, using external iteration. It's not as elegant perhaps, but it's more pragmatic IMO.
The next question is - what does the compiler produce? My idea is that it produces pickles! More specifically a list of
PickledCase(compiled scenarios). This is a struct that is completely decoupled from the Gherkin AST - it only has asourcearray withLocationto produce a stack trace.The pickles would eventually move to a separate repo/library, and Cucumber implementations will only depend on pickles for the runtime. This would enable is to come up with alternative formats for Cucumber, such as Markdown or even Excel. Just write another parser/compiler that produces pickles.
Let me know what you think. /cc @gasparnagy @muhqu @jbpros @tooky @mattwynne @everzet @stof
The text was updated successfully, but these errors were encountered: