Make random order stable and platform-independent

features/docs/cli/randomize.feature

@@ -9,6 +9,10 @@ Feature: Randomize
   you can reproduce that run by using the seed that's printed at the end of
   the test run.
 
+  For a given seed, the order of scenarios is constant, i.e. if step A runs
+  before step B, it will always run before step B even if other steps are
+  skipped.
+
   Background:
     Given a file named "features/bad_practice_part_1.feature" with:
       """
@@ -27,6 +31,14 @@ Feature: Randomize
         Scenario: Depend on state from a preceding feature
           When I depend on the state
       """
+    And a file named "features/unrelated.feature" with:
+      """
+      Feature: Unrelated
+
+        @skipme
+        Scenario: Do something unrelated
+          When I do something
+      """
     And a file named "features/step_definitions/steps.rb" with:
       """
       Given(/^I set some state$/) do
@@ -36,6 +48,9 @@ Feature: Randomize
       Given(/^I depend on the state$/) do
         raise "I expect the state to be set!" unless $global_state == "set"
       end
+
+      Given(/^I do something$/) do
+      end
       """
 
   Scenario: Run scenarios in order
@@ -44,7 +59,7 @@ Feature: Randomize
 
   @spawn
   Scenario: Run scenarios randomized
-    When I run `cucumber --order random:41529 -q`
+    When I run `cucumber --order random:41544 -q`
     Then it should fail
     And the stdout should contain exactly:
       """
@@ -56,6 +71,12 @@ Feature: Randomize
             ./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
             features/bad_practice_part_1.feature:7:in `When I depend on the state'
 
+      Feature: Unrelated
+
+        @skipme
+        Scenario: Do something unrelated
+          When I do something
+
       Feature: Bad practice, part 2
 
         Scenario: Depend on state from a preceding feature
@@ -72,11 +93,48 @@ Feature: Randomize
       Failing Scenarios:
       cucumber features/bad_practice_part_1.feature:6
       cucumber features/bad_practice_part_2.feature:3
-
+
+      4 scenarios (2 failed, 2 passed)
+      4 steps (2 failed, 2 passed)
+
+      Randomized with seed 41544
+
+      """
+
+  @spawn
+  Scenario: Run scenarios randomized with some skipped
+    When I run `cucumber --tags ~@skipme --order random:41544 -q`
+    Then it should fail
+    And the stdout should contain exactly:
+      """
+      Feature: Bad practice, part 1
+
+        Scenario: Depend on state from a preceding scenario
+          When I depend on the state
+            I expect the state to be set! (RuntimeError)
+            ./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
+            features/bad_practice_part_1.feature:7:in `When I depend on the state'
+
+      Feature: Bad practice, part 2
+
+        Scenario: Depend on state from a preceding feature
+          When I depend on the state
+            I expect the state to be set! (RuntimeError)
+            ./features/step_definitions/steps.rb:6:in `/^I depend on the state$/'
+            features/bad_practice_part_2.feature:4:in `When I depend on the state'
+
+      Feature: Bad practice, part 1
+
+        Scenario: Set state
+          Given I set some state
+
+      Failing Scenarios:
+      cucumber features/bad_practice_part_1.feature:6
+      cucumber features/bad_practice_part_2.feature:3
+
       3 scenarios (2 failed, 1 passed)
       3 steps (2 failed, 1 passed)
-
-      Randomized with seed 41529
-
-      """
 
+      Randomized with seed 41544
+
+      """

lib/cucumber/filters/randomizer.rb

@@ -1,3 +1,5 @@
+require 'digest/sha2'
+
 module Cucumber
   module Filters
 
@@ -29,7 +31,10 @@ def with_receiver(receiver)
       private
 
       def shuffled_test_cases
-        @test_cases.shuffle(random: Random.new(seed))
+        digester = Digest::SHA2.new(256)
+        @test_cases.map.with_index.
+          sort_by { |_, index| digester.digest((@seed + index).to_s) }.
+          map { |test_case, _| test_case }
       end
 
       attr_reader :seed