Ginkgo 2.0 Proposal

[onsi]

The first commit to Ginkgo was on August 19th, 2013. Much fruitful community feedback - and code cruft - have accumulated over the years. Both can be addressed with a new major version of Ginkgo: Ginkgo 2.0.

The backlog capturing the work is here: https://www.pivotaltracker.com/n/projects/2493102

The original proposal for Ginkgo 2.0 has been pulled together as a google doc here. Note that the google doc is now stale - the Tracker backlog reflects the latest work most accurately.

V2 is in active development on the ver2 branch and an uptodate changelog and migration doc is maintained on the branch here

If you use Ginkgo and are interested in what's in 2.0 please comment on this issue!

[onsi]

This proposal has been publicly available for three weeks. After receiving some helpful feedback that has made its way into the proposal I think it's time to start development.

[onsi]

I've put together a project to track progress toward v2:
https://github.com/onsi/ginkgo/projects/2

[onsi]

The first iteration of V2 is out on the ver2 branch. There are a few new features on that branch, and most of the deprecations/removals lists in the V2 doc. I'll be adding new functionality over the coming weeks/months.

This changelog + migration doc:
https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
will be kept uptodate as I go.

I'll be removing the GitHub v2 project in favor of this Pivotal Tracker backlog that I'm using:
https://www.pivotaltracker.com/n/projects/2493102

[mbalcomm]

I just got this "warning" :
You are using a custom reporter. Support for custom reporters will likely be removed in V2. Most users were using them to generate junit or teamcity reports and this functionality will be merged into the core reporter. In addition, Ginkgo 2.0 will support emitting a JSON-formatted report that users can then manipulate to generate custom reports.

We have been using customer reporters for our kubernetes tests. Kubernetes test framework is based on Ginkgo. We have reports for storing test reports to databases that require specific data structures.
One reporter stores specific data into Prometheus. It would have to be a custom reporter.
Another reporter does generate json format results files. we could replace it with the proposed Json reporter, but it also lacks some functionalities we need. For one thing is the ability to access detailed information, other than *SpecSummary
After reading the mitrating_to_v2, I am not very clear as how we can adopt new reporters and yet still be able to customize our specific data structures. Any earlier V2 adopters out there?
Thanks.

[onsi]

hey @mbalcomm - so glad to hear that that deprecation warning caught your eye. I've been looking to get some early feedback :)

To confirm - you've read the Reporting Infrastructure section in MIGRATING_TO_V2. Correct?

For your use case, assuming you don't need to emit test information in real time, I would expect that adding

var _ = ReportAfterSuite(func(report types.Report) {
	reporters.ReportViaDeprecatedReporter(prometheusReporter, report)
	reporters.ReportViaDeprecatedReporter(otherReporter, report)
})

would be a working starting point. Please let me know if you're seeing issues with that approach.

A few additional questions:

1. You mention "For one thing is the ability to access detailed information, other than *SpecSummary" - can you say more? It's possible what's currently on V2 will work for your usecase. It's possible you'll need a few more pieces that are currently planned for the V2 release which I'd be happy to describe to confirm they'll serve your usecase.
2. If it's publicly accessible, can you point me to the test suites and the custom reporters that you're referring to?
3. Would you be up for trying out V2 to give me some real-life feedback on this custom reporting piece?

Thanks!

[mbalcomm]

@onsi Thanks for your responses!
Yes, I have read the Migration_to_V2. and to answer your question:

1. The specific information I would like to capture in the report is "By". We use By() inside It for detailed test run information.
2. One reporter we are using is Allure Report(https://docs.qameta.io/allure/) since it does not have Golang plugin, we have to write our own. We like the Allure reporter's Step and attachment features. Allure Reporter is an open-source project. Although Kubernetes is open-source, currently our tests are not.
3. I would be happy to try V2 out.

[onsi]

Hey @mbalcomm:

Thanks for your reply. Some thoughts:

1. The specific information I would like to capture in the report is "By". We use By() inside It for detailed test run information.
   How are you currently capturing the information in By. Currently, By simply writes to the GinkgoWriter but I'm thinking of evolving it to better integrate into the new reporting infrastructure which would make it easier to post-process programatically.

At the same time, the main reason that By exists is because it's hard for folks to write small granular It tests when their test cases are large and complex. V2 will be introducing ordered contexts to make it possible to break out large complex Its into a collection of smaller Its that are guaranteed to run in order.

In any event, the behavior of By hasn't changed yet and when it does it will become better integrated into the new reporting infrastructure.

2. One reporter we are using is Allure Report(https://docs.qameta.io/allure/) since it does not have Golang plugin, we have to write our own. We like the Allure reporter's Step and attachment features. Allure Reporter is an open-source project. Although Kubernetes is open-source, currently our tests are not.

Have you written a custom reporter to emit to Allure? If so that should continue to work via reporters.ReportViaDeprecatedReporter. If there's enough community interest I could also imagine making Allure's native XML format a first-class report target for Ginkgo so you would simply run ginkgo -allure-report=out.xml but that's not something I'm ready to prioritize yet.

The other piece of reporting infrastructure that is coming in V2 will be to support things like attachments. So users can do stuff like:

It("...", func() {
    ....
    Report("key", value)
})

Where value can be any object. Ginkgo will include the passed in key:value pair in the generated report allowing it to be exported as desired.

3. I would be happy to try V2 out.
   Great! It's all on the ver2 branch.

[jakeczyz]

Quick Comment:

We are using a custom reporter and not ready to migrate to v2 until v2 is out of beta/stable. So, in the mean time, whenever we run tests we get those several lines of deprecation warning text eating up screen space (and attention).

It would be super helpful if there was a way to disable the deprecation warnings via flag or env var or otherwise. :)

[onsi]

hey @jakeczyz PTAL at #802 - you can now set an env var to silence deprecations

[bobdonat]

I am experimenting with v2 and I am interested in the test label feature. It appears that this feature is not there yet. Is this feature still planned for v2 and when will that feature be available?

[onsi]

Yep - still planned. Summer schedules have slowed down development a bit so i'm having a harder time reasoning about the timeline but i'm hoping to get to that feature in the next couple 2ish months.

[chrisdoherty4]

We were using custom reporters to send information to TestRail. It looks like the ReportAfterSuite() capability will satisfy our needs though - thought I'd give the feedback none-the-less.

[onsi]

thanks @chrisdoherty4 !

[TaylorOno]

After the deprecation custom reports will the any built in reports allow for integration with Goland which seems to only play nice with the go test format?

Reports like https://github.com/SemanticallyNull/golandreporter which provide real time feedback

[onsi]

hey @LittleFox94 - thanks for sharing this. Indeed the semantics for AfterEach is to run after every It and the way this plays with nested Ordered containers can be a bit confusing. In fact, nesting Ordered containers - while possible - can definitely have some confusing behavior. It's certainly added a lot of complexity to the parts of Ginkgo's codebase that need to handle the various edge cases that come up!

I could imagine changing things around so that Ordered containers act more like large its and force BeforeEach and AfterEach nodes at a shallower nesting level to behave like BeforeAll and AfterAll nodes within the Ordered container - but I worry that will make things harder to reason about.

In this particular case I'd suggest avoiding using a nested Ordered container. A few suggestions (and I'd be happy to think about it a bit more deeply with you if you can share more detail and/or point me at the repo with this code):

1. Replace the nested Ordered container with an It - there doesn't seem to be too much going on in the container so you could:

Describe
|-- BeforeEach(create object of another API we need for our tests)
|-- AfterEach(delete object created in BeforeEach)
|-- It("successfully creates and deletes a fresh object")
|-- Context("working with an existing test Object")
| |-- BeforeEach("create test Object")
| |-- AfterEach("delete test Object")
| |-- It("finds Object when listing")
| |-- It("retrieves Object with expected values")

2. Pull out the ordered context into its own top-level container:

Describe
|-- BeforeEach(create object of another API we need for our tests)
|-- AfterEach(delete object created in BeforeEach) // problematic AfterEach 
|-- Context("working with an existing test Object")
| |-- BeforeEach("create test Object")
| |-- AfterEach("delete test Object")
| |-- It("finds Object when listing")
| |-- It("retrieves Object with expected values")

Describe("working on a fresh object", Ordered)
|-- BeforeAll(create object of another API we need for our tests)
|-- AfterAll(delete object created in BeforeAll) 
| |-- It("successfully creates the object")
| |-- It("successfully deletes the object")

3. Make the entire container ordered:

Describe("...", Ordered)
|-- BeforeAll(create object of another API we need for our tests) //assumes that the object can be reused across all these tests
|-- AfterAll(delete object created in BeforeAll) 
|-- Context("working on a fresh object")
| |-- It("successfully creates the object")
| |-- It("successfully deletes the object")
|-- Context("working with an existing test Object")
| |-- BeforeAll("create test Object") //or keep as a BeforeEach - depends on your setup
| |-- AfterAll("delete test Object")
| |-- It("finds Object when listing")
| |-- It("retrieves Object with expected values")

[onsi]

@LittleFox94 i've thought about it more and think your intuition around how outer BeforeEach and AfterEach should behave for Ordered containers seems more and more right to me. I'm going to see how hard/easy it would be to have BeforeEach and AfterEach at shallower nesting levels run just once around Ordered nodes.

Would love to hear from you and others on this thread if folks have any thoughts/opinions.

@LittleFox94 in your case would it work for the BeforeEach to run just once for that nested Ordered context? I know you'd originally had questions about just the AfterEach...

[LittleFox94]

first: sorry for not having answered to your first reply, that task was pushed aside for now and the notification email still in inbox to process at some point ^^'

In my case BeforeEach and AfterEach each running once for the Ordered container would work great and I think they should always work as pair with same behavior. I'm not sure though if changing the semantics of Before/AfterEach is good or if adding new hooks or optional parameters might be better - feels like a kinda huge semantic change to me

[onsi]

Thanks. I don't think it's a super huge change since Ordered containers are a new construct and really behave like large Its composed of more granular subunits.

[LittleFox94]

Hm yes, that makes sense

[kgadams]

Hello, I am looking for a way to integrate some reporting about the progress of Ginkgo into our own log file.
Right now we get ginkgo messages on stdout (including the configurable error reporting) and logging from our own coding in a dedicated logfile.
Now I would like to add logfile entries about all the actions that ginkgo takes, so that I can match this more easily.
I had started to implement a customer reporter for this, as the hooks in the interface look promising. Now I find that this is being deprecated.
Is there another mechanism in Ginkgo V2 that will enable this functionality? The migration guide did not give me any hints.

[onsi]

hey @kgadams - the topic of V2's reporting infrastructure is covered in the migration guide here.

In brief: you can probably accomplish what you want using ReportBeforeEach and ReportAfterEach. However if you intend to run your specs in parallel and want to emit logging information into a single global log-file you will run into issues with interleaved output. There are several strategies to manage this that I'd be happy to get into - but we should probably open a different issue for that.

Lastly - the new V2 documentation goes into the reporting infrastructure in a bit more depth than the migrating guide. You can check that out here.

[kgadams]

In brief: you can probably accomplish what you want using [ReportBeforeEach and ReportAfterEach]

Ok, thank you, that looks good to me.

Lastly - the new V2 documentation goes into the reporting infrastructure in a bit more depth than the migrating guide. You can check that out here.

Thanks again 👍

[onsi]

hey @LittleFox94 I changed tack and ended up leaving the default seating of *Each untouched. You can, however, now decorate BeforeEach and AfterEach with the new OncePerOrdered decorator. Please bump to the latest and give it a try!

[onsi]

Hey all - we're super close to the GA here. I just pushed out one last release candidate. rc3 is now out and includes the new OncePerOrdered decorator. If you've been using the 2.0 RCs THANK YOU! The feedback has been invaluable! Please give this latest release candidate a try (especially if you're using Ordered containers).

I hope to have the 2.0 GA out before the new year!

[LittleFox94]

hey @LittleFox94 I changed tack and ended up leaving the default seating of *Each untouched. You can, however, now decorate BeforeEach and AfterEach with the new OncePerOrdered decorator. Please bump to the latest and give it a try!

looks good to me, I like this solution :)

[onsi]

🎉🎉 🎉

After many months - and just before we all ring in the new year - I'm super happy to share that Ginkgo 2.0 is now officially GA. The v2.0.0 code is now tagged and on master. Huge thanks to the many users who gave the betas and RCs a try and provided so much valuable feedback to make this release what it is today.

🎉🎉 🎉