Python SyntaxError: EOL while scanning string literal

[pheeria]

Hi 🖖

Thanks for the awesome plugin! And for the recent Python support, looks like I don't need vim-repl anymore 😄
I've found the following issue while using Conjure with Python.

This code runs as expected:

"hey" + "\n" + "ho"

### Output

# eval (root-form): "hey" + "\n" + "ho"
'hey\nho'

But putting this concatenation in a function fails:

def func():
    return "hey" + "\n" + "ho"

### Output

# eval (root-form): def func(): return "hey" + "\n" + "ho"
# (out)   File "<stdin>", line 1, in <module>
# (out)   File "<string>", line 3
# (out)     return "hey" + "
# (out)                    ^
SyntaxError: EOL while scanning string literal

The same is also true for using join:

"\n".join(["hey", "ho"])

All of the above code runs when executed from console.
From my very limited understanding escape-strs from fnl/conjure/client/python/stdio.fnl should not escape already escaped characters. Please let me know if I can help further.

nvim: 0.8.0
conjure: latest master, 8.10.22

[Olical]

Thanks for the report! What's fun is the log is telling us Conjure selected all the right code with tree sitter:

# eval (root-form): def func(): return "hey" + "\n" + "ho"

The issue is probably some escaping that's required when sending Python code across this stdin stream. Like the \n needs to become \\n I think.

[Olical]

Current investigation has revealed that Python's exec() which we need to rely on to be able to evaluate your code correctly doesn't like \n character's in it's source code. So \n in a normal Python source file is interpreted differently when sent into a REPL through the exec() call.

If I don't use exec() then your REPL will hang and throw errors if you try to eval loops / function definitions etc that require very specific sequences of tabs and newlines (and won't allow two new lines in a row!). So I think I either need to overhaul how the Python REPL works, use a completely different Python REPL or maybe find a way to pipe your code into a /tmp file and then load that in the REPL.

If I do that though you'll lose state between your evals, so I'd need to find a way to:

• Write your code to eval into a tmp file.
• Tell Python to load and eval that file.
• Somehow extract the context of that eval into your current REPL env so the defined functions etc are shared between this eval and the next.

Maybe there's a Python function that lets me run a file of Python code and capture the scope for future reference 🤔

Just sharing my findings here really, this seems tricky because Python's REPL makes it REALLY hard to just pipe code into it. It's intended for humans to use, not other programs, which really sucks. We need to find a way to configure it to play nicely with another program piping things into it. The current workarounds that deal with these problems cause this \n related bug 😢

[pheeria]

That's interesting! I was about to add, that \t mysteriously worked. That explains 👍
And \\n works, that's how I continued (I need to change it when using as a script, but that's ok).

[russtoku]

I've been trying to work on this problem. Unfortunately, no success. I was initially OK with having to change \n to \\n while in Conjure and undoing when running the file as a script but it's bothering me more. Ther escape-strs function in the Python client seems to only deal with double quotes within a string of Python code.

[russtoku]

The problem with using exec when sending multiple lines to the stdio Python client has to do with the child process that the Python repl is made to run in. I've spent a week trying to see if the bracketed paste mode would work and can confirm that it doesn't with the current client. I've noted this on #451 (comment).

[Olical]

I think I've fixed this on the develop branch, let me know!

[russtoku]

Wow! I tested the change on the dev/python/sandbox.py file and four of my Python scripts that use modules like BeautifulSoup, requests, etc. Seems to be a good fix! Thanks!