Notify syscall 180709 #457

craigphicks · 2018-07-10T11:28:21Z

Send Notifications by arbitrary program via python systemcall

The [isso] repository was forked by github user craigphicks. A branch notify-syscall-180709 was created. The branch is described herein.

The main uncertainly I have as author is the use of subprocess.run which appeared in Python3.5. Which versions of Python must be supported? Must Python 2 be supported? (See Limitations at the end of this document).

This comment is copied from the file ./README-notify-syscall-180709 in hte main directory.

Motivation

Security

When isso uses the SMTP notification backend, it also sends the username and password. The are various perspectives from which this could be problematic.

Password in clear: In the worst case TLS (Transport Layer Security) is not established before passing the password and so the password is transmitted in the clear.
Key better than password: Even with TLS, the password is still in the clear at either end. While this is usually OK, security can be improved by using a key which can be periodically refreshed. The point of the key is that is that it is only authorizes a narrow range of actions. Especially, it does not allow logging in as mail account owner, which could be very damaging.

Dependence on Python SMTP implementation

Isso is tied to the Python SMTP implementation. Even if SMTP is the desired protocol, the may be other software besides the Python SMTP implementation which gives a better result, e.g. in terms of enabling TLS or working with tokens.

Some examples of alternate software are the multitude of generic sendmail programs, e.g. sendmail, postfix, exim, nullmailer, all of which all share the linux sendmail application interface as a standard.

Also there utilities such as curl and swaks which are very flexible with many options. In case something is not working it is almost always possible to work through it with these programs (or change to another program). Although they are not designed for production use, they are sufficient on the usage scale expected by a typical isso installation.

Solution

Outline

The isso administrator specifies the program and it's arguments in the isso configuration file. When a comment notification event occurs the specified program and arguments are executed via a system call. The message is passed to the standard input of that program.

Administrator interface

Add syscall to the list of notify backends:

[general]
...
notify = stdout, syscall
...

Create a section [syscall] and therein specify the program and its arguments. The following example shows curl being used to transmit an email request through a (free) Mailgun account:

[syscall]
call = /usr/bin/curl
 --trace-ascii
 /tmp/mg1.log
 --user
 api:key-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 https://api.mailgun.net/v3/example.com/messages
 -F
 [email protected]
 -F
 [email protected]
 -F
 subject=isso {{SUBJECT}}
 -F
 text=<-

The key call appears on the first line followed by equal. The value is split between lines with exactly one argument per line. Each successive value line must have an empty space at the beginning.
Often in a shell command line quotation mark must be used to prevent a parser from interpreting spaces as being argument delimiters. E.g., for the subject

 subject=isso {{SUBJECT}}

a bash script would require quotes placed like "subject=isso {{SUBJECT}}" or subject="isso {{SUBJECT}}" to keep the argument together. The shell would interpret the quotes for their syntactic meaning, and then remove them before passing each argument phrase to the program as, e.g. argv[11] or equivalent.

However, for the isso configuration call value above, quotes are not required because the grouping is done by lines. In fact quotes for the purpose of syntactic grouping must NOT be used. The quotes will not be removed before passing to the program, and the input will likely be invalid. Quotes can still be used as part of the content of any argument, as required.

Some (but not all) scenarios require the mail subject to be specified as part of the arguments, and not in the standard input data. The above curl example shows such a case. In order to enable variable subject line content a template parameter %SUBJECT% is provided. The isso software will render this template %SUBJECT% as the fixed formula:

...<last 15 characters of blog uri> ::: <first 15 characters of comment>...

The last part of the last curl argument: `<-' , is a curl specific instruction to accept data from standard input. Because the message body is variable it must be transmitted via standard input to the process. So any program used must be configured (or already be hard coded) to accept standard input.

Be aware that protocol issues depend on the called program and the site receiving the data. It is not handled by the isso code. For example, although the above example appears to transact via HTTPS, there is some possibility it might downgrading to HTTP after authentication for the message transaction part. (The log file is still under interpretation.)

Read the current limitations section at the end of this document.

Implementation

The notify-syscall-180708 branch adds a new module Syscall in the isso/ext/notifications.py file.

It formats the mail message body in the same way as the existing SMTP module, but without the SMTP header and quopri / base64 encoding.

The command and arguments as specified in the configuration are supplied as a list which is the first parameter of subprocess.run.

The message body is encoded to binary and passed as standard input to the invoked process.

When complete, subprocess.run returns an object with members returncode, stdout, and 'stderr'. All are logged via the global logger (usually to /var/log/isso.log).

Isso expects that success is indicated by a returncode value of zero, and anything else is error. However, an error does not result in an isso program error. The only difference in isso program behavior is that for a non-zero result isso will log via

logger.error(....)

and for a zero result via

logger.info(....)

Other changes

The notification backend Stdout module was rewritten to fix a bug caused by trying to log a bloom filter as text. Also, the relation between function names, log content, and events was cleaned up.

IMPORTANT: Current Limitations

Implementation

subprocess.run is not available before Python 3.5. Certainly not in Python2. In environments where the code will not run it should not be enabled in the configuration file, (if it even compiles).

blatinier · 2018-07-28T20:54:25Z

isso/config.py

@@ -135,6 +135,8 @@ def setify(cp): return set((section, option) for section in cp.sections()
        parser.read(user)

    for item in setify(parser).difference(a):
+        if item[0]=="syscall":


spaces around operator

blatinier · 2018-07-28T20:55:46Z

README-notify-syscall.md

@@ -0,0 +1,114 @@
+


Could you add this in the docs folder and link it so it appears on the website when @posativ will have time to update stuffs.

blatinier · 2018-07-28T21:02:04Z

isso/ext/notifications.py

+    if comment["email"]:
+        author += " <%s>" % comment["email"]
+
+    rv.write(author + " wrote:\n")


Use string formatting instead of concatenation. (it applies for all the function below :) )
Also calling only once rv.write for subsequent calls would be better.

blatinier · 2018-07-28T21:03:59Z

isso/ext/notifications.py


    def _delete_comment(self, id):
-        logger.info('comment %i deleted', id)
+        logger.info("comments.delete %i ", id)


Remove the trailing space in the string

blatinier · 2018-07-28T21:04:52Z

isso/ext/notifications.py


    def _activate_comment(self, id):
-        logger.info("comment %s activated" % id)
+        logger.info("comments.activate %s" % id)


Do not use string formatting here, let logging do it. It will compute strings only if needed according to the log level.

I saw there's more above of this too to correct

blatinier · 2018-07-28T21:10:10Z

isso/ext/notifications.py

+        msgbody=_format(thread,comment,self.general_host,key)
+        subject = "..." + thread['uri'][-15:] + " :: " + comment['text'][:15] + "..."
+        cmdlist = [ a.replace('{{SUBJECT}}',subject,1) for a in self.conf.getiter('call') ]
+        logger.info(cmdlist);


useless semicolon

blatinier · 2018-07-28T21:11:11Z

isso/ext/notifications.py

+        key=self.isso.sign(comment["id"])
+        msgbody=_format(thread,comment,self.general_host,key)
+        subject = "..." + thread['uri'][-15:] + " :: " + comment['text'][:15] + "..."
+        cmdlist = [ a.replace('{{SUBJECT}}',subject,1) for a in self.conf.getiter('call') ]


No space after opening bracket.
No space before opening bracket.
Spaces after comma.

blatinier · 2018-07-28T21:11:37Z

isso/ext/notifications.py

+        subject = "..." + thread['uri'][-15:] + " :: " + comment['text'][:15] + "..."
+        cmdlist = [ a.replace('{{SUBJECT}}',subject,1) for a in self.conf.getiter('call') ]
+        logger.info(cmdlist);
+        p=subprocess.run(cmdlist,


spaces around assignement operator

blatinier · 2018-07-28T21:12:10Z

isso/ext/notifications.py

+        + "stdout:\n" \
+        + str(p.stdout) + "\n" \
+        + "stderr:\n" \
+        + str(p.stderr) + "\n"


Use string formatting. And maybe multiline string using triple quotes?

blatinier · 2018-07-28T21:13:17Z

isso/ext/notifications.py

+                         input=str(msgbody).encode(),
+                         stdout=subprocess.PIPE,
+                         stderr=subprocess.PIPE)
+        s = "Syscall: return code " + str(p.returncode) + "\n" \


Name it logged_string or something more significant.

blatinier · 2018-07-28T21:20:01Z

Ok for now I checked the code and it seems ok. Some minor changes (even though there's a few).
Please rebase too.

Important stuff to check:

You read general → host try to see if public-endpoint introduced recently could fit in (needs to rebase first)
Add the README in the docs folders with some link to it (check https://posativ.org/isso/docs/ to see the best place to put it)

I still need to test the PR too :)
Concerning the suported pytnon versions I think it's fair enough to enable you feature in python3.5+ only. But it should not crash if used with older version. It should log some error message to kindly suggest to move their ass into python3 :)

ix5 · 2022-04-23T23:30:55Z

Stale and unlikely to ever be merged without a massive rebase.

craigphicks · 2022-04-24T20:19:30Z

I should have closed this pull long ago - apologies. There really isn't any great need for the suggested feature - what exists is perfectly satisfactory.

root added 5 commits July 9, 2018 20:28

allow notifications via system call to arbitrary program

5955df3

add README-notify-syscall.md to document this branch

7a2923d

minor modifications

006d720

one call arg per line; {{SUBJECT}} template

058e68d

update doc

45fb221

craigphicks mentioned this pull request Jul 11, 2018

Which Python versions must Isso support? #459

Closed

craigphicks added 2 commits July 11, 2018 02:32

clean up code

8b3dc01

fix mail links for activate/delete

7dfa811

blatinier requested changes Jul 28, 2018

View reviewed changes

Konzertheld mentioned this pull request Jul 12, 2019

Possibility to use Local Sendmail? #346

Closed

ix5 mentioned this pull request Dec 22, 2021

Please add more maintainers #752

Closed

ix5 mentioned this pull request Feb 10, 2022

XMPP notifications #31

Closed

ix5 closed this Apr 23, 2022

craigphicks deleted the notify-syscall-180709 branch April 24, 2022 17:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Notify syscall 180709 #457

Notify syscall 180709 #457

craigphicks commented Jul 10, 2018 •

edited

Loading

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier Jul 28, 2018

blatinier commented Jul 28, 2018

ix5 commented Apr 23, 2022

craigphicks commented Apr 24, 2022

Notify syscall 180709 #457

Notify syscall 180709 #457

Conversation

craigphicks commented Jul 10, 2018 • edited Loading

Send Notifications by arbitrary program via python systemcall

Motivation

Security

Dependence on Python SMTP implementation

Solution

Outline

Administrator interface

Implementation

Other changes

IMPORTANT: Current Limitations

Implementation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blatinier commented Jul 28, 2018

ix5 commented Apr 23, 2022

craigphicks commented Apr 24, 2022

craigphicks commented Jul 10, 2018 •

edited

Loading