TextUI runner command line options should override configuration file

[jhansche]

<phpunit>

  <groups>
    <exclude>
      <group>FlakyTests</group>
    </exclude>
  </groups>

  <testsuites />
</phpunit>

In this example any test case annotated with @group FlakyTests would be ignored when running phpunit in the same directory. However, if I then want to run just the FlakyTests in a separate execution, the common-sense way to attempt that would be:

$ phpunit --group FlakyTests

however, that does not execute the FlakyTests group. Instead, it seems to try to combine both the --group (i.e., <include/>) and <exclude/> (i.e., --exclude-group) options, and it ends up running nothing -- it tries to run only the FlakyTests group, but because it's being excluded by the xml file, it refuses to do so, and therefore does not run anything. The only way then to override that behavior so that you run only the originally-excluded tests is by excluding a different group (doesn't matter if it exists, just something other than the "FlakyTests" that we want to run):

$ phpunit --group FlakyTests --exclude-group nonexistent

The --exclude-group is extraneous, and not immediately obvious that it would be required at all (and is only a workaround). A user would normally think that a command-line option would always override an XML directive, and the thing is, it does override the XML, but the problem is that a command-line --group does not override an XML exclude.

It's not a huge deal, since it can be worked around, as mentioned above. But it is a strange requirement. If the tests are being run in a CI server, it's even easier to configure the job to override the exclusion, or provide a secondary configuration file, or even use an xslt ant task to first strip the exclusion from the configuration file. I just thought it was a weird behavior. I would consider something given on the command-line to be the final input that nothing else should be able to override E.g., I would consider the opposite to be just as strange: <include>EasyTests</include> exists in the XML file, but --exclude-group EasyTests given on command line. In other words, it seems obvious to me as a user that whatever I pass on the command line should always override any conflicting statements in the XML file.

[edorian]

Referenced issue #322 has another reproduce

[HelloGrayson]

+1

I've written unit tests for a class that interacts with AWS (rather slow api) using stubs. I've also written them using full functional testing. The functional tests take about 10 minutes to run. I've put these into an aws group and added them to the exclude section - when something goes wrong in my application I wan't to be able to run the full functional tests to determine the issue where stubs couldn't.

Supporting this given the current feature set is rather ugly as you end up doing something like so...

http://stackoverflow.com/questions/2094970/passing-parameters-to-phpunit

[HelloGrayson]

I've determined a workaround for those interested in fixing this phpunit override group issue.

<groups>
    <include>
        <group>__nogroup__</group>
    </include>
</groups>

Running phpunit with the above will run tests for all tests that do not have a group. To run the "excluded" group, you can now simply do:

phpunit --group aws

If you need more groups to be ran by default, then add additional <group> elements to the include element.