Roadmap and Challenges

Currently, Rambo is able to generate tests for a basic API with a GET route on a single collection. It needs additional features to bring it to an MVP. This page outlines some of the upcoming challenges.

Testing

Current Rambo tests involve comparing Rambo output to fixture files. However, the latest version of the JSON Test Data gem throws a wrench in this approach by generating data randomly. We will need to find a more robust way to test Rambo that either doesn't rely on specific data values or stubs the output of JSON Test Data.

Although there is comprehensive test coverage testing that Rambo generates the expected output, there are currently no tests that verify the generated tests actually work properly. We will need to determine a way to test this output.

Generation of Test Data

It is unclear at this point how we will generate the appropriate data for testing each API. Although generating request data is not particularly problematic, creating the system state that will lead to particular responses will be less trivial. A non-exhaustive list of the specific challenges posed here includes:

• URI Parameters: Given a resource called /users and a route GET /users/:id, what value should be used for the :id param, as this would internally depend on which users exist in the system?
• Response Bodies: How will Rambo test that appropriate response bodies are returned, without knowing what resources are available to the system? Should Rambo-generated tests validate that the data returned by an endpoint are the correct ones, or should it simply validate that they match the schema? If the latter, what should Rambo do in the case that no schema is given and only an example is available? (Currently, Rambo does test that the response matches a schema.)

Handling Nested Resources

Rambo should be able to use recursion to handle nested resources, however, implementing this will depend on first solving the problem outlined above with generating valid URI parameters.

Handling Multiple Routes

Although Rambo does generate tests for multiple routes, this functionality is not tested and needs to be expanded. Additionally, there should be the option to create multiple spec files for different routes or resources, instead of just one long one.

Handling Multiple Request or Response Schemas

RAML allows for multiple possible requests or responses to be defined for a given resource, whereas currently Rambo only supports testing a single possible request or response.

Generating Data when Properties Depend on Each Other

Rambo uses the JSON Test Data gem to generate the request bodies that are sent to an endpoint when the Rambo tests are run. One problem I've run into is that, in some cases, different values in the request bodies depend on each other. For example, acceptable values of a "state" attribute might be dependent on the value of the "country" attribute. Changes will need to be made to either Rambo or JSON Test Data to account for this. (I am the author of JSON Test Data, so I can assure you that any PRs will be reviewed promptly.)

Resources

There are options with JSON Schema to handle this contingency. An example can be found here. There is also an open issue in the JSON Test Data repo addressing this.