Document or automate stress-testing for our hubs

[choldgraf]

Background and proposal

Context
Most of our clusters have the ability to scale up or down gracefully when there is enough demand. This also maps on to particularly important times of user activity, because it usually means a lot of people are paying attention to the hub at once.

It should be possible to write scripts that simulate these "group log on" activities, to see how the infrastructure responds.

Proposal

I propose that we build infrastructure and/or document a process for stress-testing our hubs. This would let us give communities more assurance that things will work as expected, and might also help us catch bugs before they're big fires.

ref: #1032 where this issue came up as well

Implementation guide and constraints

I'm not sure what's the best way to do this, though I know that @yuvipanda has worked on this kind of thing at Berkeley before, or maybe we've tried something like this for mybinder.org?

Updates and ongoing work

No response

[damianavila]

There is some previous discussion (an interesting links) here: #883 (comment)
IIRC, there is also a project from @yuvipanda: https://github.com/yuvipanda/hubtraf, but not sure the status on that one (no recent changes in the codebase).