How to run gomacro inside an existing application [solved]

[cdevr]

Ideally with the application's functions available. Do you think that can be done ?

[cosmos72]

Yes to both questions :)

Running gomacro inside an existing Go application is very easy:

package main
import (
	"fmt"
	"reflect"
	"github.com/cosmos72/gomacro/fast"
)
func RunGomacro(toeval string) reflect.Value {
	interp := fast.New()
	// for simplicity, only collect the first returned value
	val, _ := interp.Eval1(toeval)
	return val
}
func main() {
	fmt.Println(RunGomacro("1+1"))
}

For performance, you'd better reuse the *fast.Interp object between calls.

Starting an interactive prompt (a REPL) is only slightly more complicated - tell me if you need instructions for that.

---

To inject your application's function into gomacro, write an init() function that adds them to imports.Packages[] - see for example https://github.com/cosmos72/gomacro/blob/master/token/x_package.go

You can either write such an init() function manually or, better, have gomacro itself generate it:
from gomacro REPL, type:

import _i "import/path/for/your/application"

It will create a file x_package.go inside your application source code with an init() function that injects all your application's types, functions, methods, constants and variables into gomacro import database.
In this way, after recompiling your application, passing the string import "import/path/for/your/application" to fast.Interp.Eval() will succeed, and all your application's functions, types, variables and constants will be available inside the interpreter

P.S. if you distribute your application, remember that Gomacro has LGPL license and you must write/tell that to your users, adding where they can download Gomacro source code from.

UPDATE license is now the more relaxed MPL 2.0

[cosmos72]

I hope the answer is sufficiently clear and detailed.

Closing the issue, feel free to reply if you need further clarifications, and I will reopen it.

[cosmos72]

Reopening to make it more visible to new users

[titpetric]

Hey, first off, thanks for a great software package! Secondly: how would you go about just invoking a function in the interpreter, that would take some go-type (interface, actually) as a parameter (and likely returns something). For example:

interp := fast.New()
interp.Eval(code)
// ...
interp.DeclVar("something", r.TypeOf(io.Writer).Elem(), new(bytes.Buffer))
interp.Eval("return fnName(something)")

Am I going in the correct direction? I'm missing something which would be basically an utility function for a function call, something like interp.Call(functionName string, ...interface{}) []r.Value, or something along those lines. Eval itself makes it a little bit hard to do the translations from Go-land into the interpreter and back, but it's great to provide the whole state up to the point of running something in there :)

[cosmos72]

Hi,
there are two ways. They both start by declaring a function in the interpreter:

interp := fast.New()
interp.Eval(`
  import "io"
  func myWrite(writer io.Writer, data []byte) (int, error) {
    return writer.Write(data)
  }`)

The first technique then calls the function with Eval as you wrote in your example:

interp.DeclVar("w", nil /*infer type*/, new(bytes.Buffer))
interp.DeclVar("b", nil /*infer type*/, []byte{1,2,3})
interp.Eval(`myWrite(w, b)`)

The second technique instead extracts the function from the interpreter and uses it as any other (compiled) function:

funv := interp.ValueOf("myWrite") // returns a reflect.Value
fun := funv.Interface().(func (io.Writer, []byte) (int, error)) // type assertion. alternatively, one can use funv.Call(...)
n, err := fun(new(bytes.Buffer), []byte{1,2,3}) // normal function call

Actually, the function Interp.ValueOf(name string) reflect.Value used in the second technique is the preferred way to retrieve constants, variables, and functions declared in the interpreter.

[glycerine]

gomacro continues to amaze me.

Is there any way to have the import _i "import/path/for/your/application" part run under go:generate so that my x_package.go imports don't get stale? update: I'll open a separate ticket for this idea.

[glycerine]

Is there an inverse of func (ir *Interp) ValueOf(name string) (value r.Value)?

That is, in my host code I have say a Go struct value (of a type precompiled into in x_package.go), and now I want to inject that value into the gomacro interpreter global variables without having the serialize it into a string and use Eval. Create a new global variable inside gomacro with the value pointing to (or a fast binary copy of) the value from the host.

update: another way of asking this would be: if I want to add x long after init() time, and I do imports.Packages[mypg].Binds["x"] = reflect.ValueOf(x), do I need to do any other call to make the interpreter take notice--like the lazy merge? I suspect there must be because I'm getting

panic: repl.go:7:32: undefined identifier: x

[cosmos72]

The content of imports.Packages[mypkg] is loaded into the interpreter when you evaluate import "mypkg". Later modifications to imports.Packages[mypkg] are not picked up.

You can instead use Interp.DeclVar(name string, typ xreflect.Type, value interface{}) to declare a global variable - if you pass a nil type, it will be inferred from value.
Note that value will be (shallow) copied, as per normal Go argument passing of interface{} parameters: if you want later modifications from one side (compiled Go) to be visible from the other side (interpreter) and vice-versa, you need to pass a pointer as value.

If you need to exchange a lot of different values between the interpreter and compiled Go, it's more efficient to use Interp.DeclVar() once with a dummy value, then call Interp.ValueOf() on the declared variable - the returned reflect.Value is settable and addressable, i.e. you can do:

interp := fast.New()
interp.DeclVar("foo", nil, int(0)) // int(0) is just an example - you can use any type
value := interp.ValueOf(foo)
// and then either
value.SetInt(5)
// or get the address once with
addr := value.Addr().Interface().(*int)
// and then perform many (fast!) pointer dereferences to inject different values:
*addr = 5
// ...
*addr = 6
// etc.

[glycerine]

This is so cool. Thanks @cosmos72.

I was able to inject a global x with interp.DeclVar(). But to inject x into package y, so far I did need to have a call to

imports.Packages["github.com/blah/long/path/to/my/y"].Binds["x"] = reflect.ValueOf(x)

prior to my

interp.DeclVar("github.com/blah/long/path/to/my/y/"+"x", nil, int(0))

call. I'm not sure if this is intended? I tried

interp.DeclVar("github.com/blah/long/path/to/my/y/"+"x", nil, int(0))

without luck. Let me know if there is a shortcut way that uses DeclVar() alone.

Thanks again.

update: arg. sorry -- my updates to the package variable aren't working. It is always that initial value.

[cosmos72]

if the global variable is in a different package, you cannot directly DeclVar it, because DeclVar creates a variable in the current package (and does not accept / in the variable name). Instead, you need to either inject the variable into imports.Packages before importing the whole package in the interpreter:

imports.Packages["github.com/blah/long/path/to/my/y"].Binds["x"] = reflect.ValueOf(&x).Elem() // to make the reflect.Value addressable and settable

or to import the package (if not done already), change to that package, declare the variable, then change back (more verbose):

interp.ImportPackage("y", "github.com/blah/long/path/to/my/y"`) // equivalent to interp.Eval(`import y "github.com/blah/long/path/to/my/y"`)

interp.ChangePackage("y", "github.com/blah/long/path/to/my/y")

interp.DeclVar("x", nil, x) // here x is interface{}, not reflect.Value - it will be copied

interp.ChangePackage("main", "main") // or to whatever was the current package

In both cases, to extract the (settable) variable x from package y you need to use the usual Go syntax pkg.symbol, i.e. y.x, which is not supported by Interp.ValueOf() - you need one of the two functions

Interp.Eval1(string) (reflect.Value, xreflect.Type)
Interp.Eval(string) ([]reflect.Value, []xreflect.Type)`

as shown below:

vaddr, _ := interp.Eval1("&y.x")
addr := vaddr.Interface().(*mytype)

Since you are trying to exchange data between compiled Go and the interpreter, the very commented examples in gomacro/_example/earljwagner1/earljwagner1.go and gomacro/_example/earljwagner2/earljwagner2.go may help too.

[glycerine]

awesome. Thanks Massimiliano!

For anyone else reading, this was my final flow for updating variable x in package y. The api evolved a little and interp.Comp was needed for the import, and DeclVar can still be used if you stay inside the package. The prior setup includes having package y have a x_package.go generated for it and y is already compiled into the host code.

// one time setup -- this is all host code, not script.
var x int
imports.Packages["github.com/path/y"].Binds["x"] = reflect.ValueOf(x)
interp.Comp.ImportPackage("lname", "github.com/path/y")
interp.ChangePackage("lname", "github.com/path/y")
interp.DeclVar("x", nil, x) // x will be copied
vo := interp.ValueOf("x")
pXinsideGomacro := vo.Addr().Interface().(*int)

// ...then to update x inside gomacro each time (still host code)
*pXinsideGomacro = x

[cosmos72]

Good :)

Some minor considerations:

• I added a method Interp.ImportPackage() so that users do not need to resort to Interp.Comp.ImportPackage() - actually, I thought the former existed already: my mistake.

• you don't need both imports.Packages["github.com/path/y"].Binds["x"] = ... and interp.DeclVar("x", nil, x) - one of them is enough, just pick what you prefer.

In case you choose the first, the reflect.Value must be settable and addressable, so you need to write it as:

// one time setup -- this is all host code, not script.
var x int
imports.Packages["github.com/path/y"].Binds["x"] = reflect.ValueOf(&x).Elem()

in this way, the variable visible inside the interpreter will be simply the compiled 'x' - any changes to it will be visible from both sides - so you can just write, for example:

// ...then to update the x visible inside gomacro each time (still host code)
x++

On the other hand, if you choose interp.DeclVar("x", nil, x), it copies the passed value instead of taking its address, so you need to explicitly retrieve the address from the interpreter, as you did.

Summarizing, there are two alternatives. They both require to generate beforehand the file x_package.go inside $GOPATH/src/github.com/path/y as described above:

============= alternative 1 =================

// one time setup -- this is all host code, not script.
interp := fast.New()
var x int
imports.Packages["github.com/path/y"].Binds["x"] = reflect.ValueOf(&x).Elem()
interp.ImportPackage("lname", "github.com/path/y")
interp.ChangePackage("lname", "github.com/path/y")

// ...then to update x inside gomacro each time (still host code)
x = someExpression()

============= alternative 2 =================

// one time setup -- this is all host code, not script.
interp := fast.New()
interp.ImportPackage("lname", "github.com/path/y")
interp.ChangePackage("lname", "github.com/path/y")
interp.DeclVar("x", nil, int(0)) // x is declared and initialized in the interpreter with a copy of `int(0)`
vo := interp.ValueOf("x")
pXinsideGomacro := vo.Addr().Interface().(*int)

// ...then to update x inside gomacro each time (still host code)
*pXinsideGomacro = someExpression()

[cosmos72]

I created the file _example/glycerine1/main.go containing the complete, commented, working code for both techniques to share variables between compiled code and gomacro interpreter.

Thanks for the useful discussion :)

[NightMachinery]

I created the file _example/glycerine1/main.go containing the complete, commented, working code for both techniques to share variables between compiled code and gomacro interpreter.

Thanks for the useful discussion :)

@cosmos72 Between this and the break statement, which one is better? Is there a big difference at all?

[cosmos72]

@cosmos72 Between this and the break statement, which one is better? Is there a big difference at all?

I did not understand the question:
are you talking about the break statement in general (and its implementation in the interpreter), or do you have a specific code example in mind?

[NightMachinery]

@cosmos72 commented on Aug 27, 2020, 12:11 PM GMT+4:30:

@cosmos72 Between this and the break statement, which one is better? Is there a big difference at all?

I did not understand the question:
are you talking about the break statement in general (and its implementation in the interpreter), or do you have a specific code example in mind?

I'm used to IPython's embed, which drops you into a REPL where you can access the local state, call functions, etc. Similar to pry, I gather. My question is, which method serves this use case better, break or the example here?

[cosmos72]

I guess you're talking about the statement "break" or _ = "break" (note the double quotes). Correct?

Then it's quite different from the example above: executing "break" or _ = "break" inside interpreted code will suspend executon and open a debugger prompt, where you can interactively access the local state, call functions, etc.
Alternatively, hitting Ctrl+C while interpreted code is running will have the same effect.
Clearly, the code you type at debugger prompt will be interpreted (not compiled - you're still inside Gomacro).

The example above instead explains how to exchange data between interpreted and compiled Go code by compiling and executing some Go code - definitely not something you can do interactively.

I hope this helps clarifying the difference.

[cdevr]

Do you think it'd be possible to go even further and run something like gophernotes from within an app? (to get a notebook interface available inside a running application)

[cosmos72]

Good question :)
it's probably better to discuss it in gophernotes - I just opened an issue there, see gopherdata/gophernotes#232