-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathREADME
174 lines (129 loc) · 6.85 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
See file COPYING distributed with the xnnppx package for the copyright
and license.
Xnnppx is a package for launching Python scripts from XNAT
(www.xnat.org).
CONTENTS
README -- This file.
COPYING -- Copyright and license.
XnatPipelineLauncher -- The drop-in replacement script for
XnatPipelineLauncher that shunts execution to a Python script,
if available.
XnatPythonLauncher -- The Python script launcher.
xnnppx.py -- The XNAT utility module for the Python script.
create_workflow_xml -- A utility program for converting a Python
or YAML dictionary into an XML pipeline descriptor for XNAT.
sample.info -- Example Python dictionary for generating an XML
pipeline descriptor.
sample.py -- Example Python pipeline script.
INSTALLATION
Xnnppx requires:
Suds: https://fedorahosted.org/suds/
pyxnat: http://packages.python.org/pyxnat/
In $PIPELINE_HOME (typically $XNAT_HOME/pipeline), move
bin/XnatPipelineLauncher to bin/XnatPipelineLauncher.orig and replace
it with XnatPipelineLauncher from this distribution. Change
pipeline_home in this (new) XnatPipelineLauncher and make sure this
file has executable permissions.
This step steals execution from XNAT and shunts it to a Python
script if it is available. Otherwise it passes control back
to XNAT's native pipeline launcher.
Install XnatPythonLauncher in $PIPELINE_HOME/bin with executable
permissions. Change the value for the pipeline config file name
in the default arguments in the definitions and defaults section.
If XnatPipelineLauncher finds a Python script that needs to be
run, it calls this script, which takes the parameters passed
by XNAT and packages them in a form easily used by the Python
script.
Install xnnppx.py in your Python path.
This is the module that the Python script will use to access
the XNAT parameters and update the workflow information.
CREATING A PYTHON PIPELINE
Create a Python script with extension .py in the same directory as
the XML pipeline descriptor (typically in $PIPELINE_HOME/catalog/.../).
XnatPipelineLauncher detects this script and launches it with
XnatPythonLauncher instead of the native XNAT pipeline launcher.
See sample.* for an example. Install sample.py in a subdirectory
of $PIPELINE_HOME/catalog, for instance
$PIPELINE_HOME/catalog/local/sample.py. XNAT will need an XML
descriptor of the pipeline in order to launch it; this can be created
by describing the pipeline as a Python or YAML dictionary (see
sample.info) and converting it with create_workflow_xml. Put the
result in $PIPELINE_HOME/catalog/local/sample.xml.
The Python script can access the pipeline arguments and parameters
in the dictionaries xnnppx.parameters and xnnppx.arguments. Because
any parameters may have more than one value, the values in
xnnppx.parameters are all lists of parameter values.
XnatPythonLauncher will redirect output to a log file (typically
in $PIPELINE_HOME/logs), update XNAT (more specifically, the workflow
XML) with the workflow execution environment and pass control to
the Python script. Once the script has started, it is then responsible
for updating the workflow XML, which can be done by calling:
xnnppx.workflow_info.update(step_id,
step_description,
percent_complete)
xnnppx.workflow_info.complete()
xnnppx.workflow_info.fail([step_description])
xnnppx.ContextManager may be used to automatically mark the workflow
XML complete using xnnppx.workflow_info.complete() or
xnnppx.workflow_info.fail() as appropriate and send a success or
failure e-mail as specified by the notify_flag (-supressNotification)
and notify_emails (-notify) arguments. See sample.py for an example.
xnnppx.send_mail(to_addrs, subject, body) can be used to send mail.
LAUNCHING A PIPELINE
For XNAT to manage a pipeline, it must first be added to XNAT, and
then to a project. As a site administrator, go to Administer ->
Pipelines, then select Add Pipeline to Repository. Give the full
path to the XML pipeline descriptor
($PIPELINE_HOME/catalog/.../sample.xml); you may leave the second
line blank. Then the pipeline needs to be added to each project
it will apply to. As a site or project administrator, go to the
project, then select Pipelines, Add More Pipelines, and Add for the
pipeline you want to add. No changes are needed to this form, but
you may choose to select the pipeline for automatic running when
data is archived to the project.
There are (at least) four ways of launching an XNAT pipeline:
automatically on archiving, through the web interface, using a REST
call, and from the XNAT server command line.
If you choose for the pipeline to be automatically launched on data
archive (see above), it will be launched automatically after the
Transfer and AutoRun pipelines have completed.
To launch a pipeline from the XNAT web interface, you will first
need to create the launching mechanism. As a site administrator,
go to Administer -> Data Types -> xnat:mrSessionData (or whatever
data type you want to work on) -> Edit, then under Available Report
Actions, set:
Name: PipelineScreen_launch_pipeline
Display Name: Launch Pipeline
Grouping:
Image: wrench.gif
Popup: sometimes
Secure Access: edit
Additional Parameters:
Sequence: 16
Now the MR Session pages will have a Launch Pipeline option in the
Actions menu; click on this and follow the prompts to start the
pipeline.
See also http://xnatdev.wikispaces.com/Pipeline+Engine.
To start a pipeline using a REST call, POST the XML for the pipeline
to be launched to
/data/archive/projects/{PROJECT}/pipelines/{NAME}/experiments/{EXPERIMENT_ID}.
The pipeline names for a project can be found using a GET to
/data/archive/projects/{PROJECT}/pipelines, and a GET to the POST
location above will give template XML that can then be edited and
used for the POST. For example:
GET /data/archive/projects/mystudy
will give us the pipelines available for the project "mystudy." If
we find one called "sample," we can then:
GET /data/archive/projects/mystudy/pipelines/sample/experiments/myxnat_E00001
to get template XML for this pipeline, and (after modifying the
parameters in the XML as needed), send it back with a POST to launch
the pipeline:
POST /data/archive/projects/mystudy/pipelines/sample/experiments/myxnat_E00001
If you have access to the XNAT server, you can launch pipelines
from the command line. XnatPipelineLauncher can be called directly;
this is useful if a pipeline fails and the full command line is
available in the logs. Similarly, XnatPythonLauncher can be called
directly if the full command line is known. XnatPythonLauncher
sends all its output to a log file by default, but if XNAT_PYTHON_NOLOG
is defined, output is not redirected and will be displayed at the
command line.