RFC : Added interrupt_workers

[amitmurthy]

Patch provides a means to support Ctrl-C for both remote as well as local workers as outlined in #3805

Base.interrupt_workers() must be called from the sigint handler as desired.

Interrupting workers has been implemented via callbacks to the ClusterManager. The system pid of the worker process is sent back as a response to the initial :join_pgrp message sent to the workers

[JeffBezanson]

Great functionality to have.

I'm a little concerned about splitting up information about a Worker so that some of it is in the Worker object, and some of it is in dictionaries in the ClusterManager. And a function accepting a symbol to tell it what to do is a little strange; seems like it should be 3 methods instead.

I think the API should be interrupt(workers()) and interrupt(n).

[amitmurthy]

I put it in the RegularCluster in order to keep it similar to Scyld, SGE and other Cluster Managers. Since the typical use cases would be ssh or local procs, I could move the required information into the Worker object and only call the ClusterManager callback if it is not of type RegularCluster. Will update the patch.

The function accepting the symbol is a callback function provided by the ClusterManager interface. Currently it is called only on 3 events - register, interrupt and deregister. There may be more events in the future - I just wanted to avoid a series of differently named callbacks. For SGE/Scyld the interrupt functionality will have to be provided by the respective ClusterManagers.

You mean interrupt(; pids = workers()) and interrupt(pid::Integer), right? I'll not export these if this functionality will only be exposed via Ctrl-C

[JeffBezanson]

Putting in a special case for RegularCluster instead of using the normal callback sounds worse to me.

[amitmurthy]

It does keep the regular code cleaner and simpler. Let me quickly try it out and we can then take a call.

[amitmurthy]

Removed dictionaries in the cluster manager to keep context. Instead, at worker creation time, the cluster managers can provide an opaque "context" object linked with each worker which is passed back in the callbacks.

The above simplifies the code a bit and also keeps the same model for RegularCluster as well as other ClusterManagers.

[amitmurthy]

Have exported interrupt.

interrupt(workers()) uses @sync and @async. I don't know if this may cause problems when called from within a signal handler as may be required for closing #3805

Do let me know if I should change it.

[amitmurthy]

@JeffBezanson - can I merge this?

[JeffBezanson]

OK, let's rebase and merge this.
I'm not sure pids needs to be a keyword argument, but that's a small detail.

[ViralBShah]

Now that this is in, it would be nice to have docs too.