Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[x/programs] Improve State-keys & Argument Passing #623

Merged
merged 78 commits into from
Dec 12, 2023
Merged
Show file tree
Hide file tree
Changes from 64 commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
c49fbc6
adding Key struct
samliok Nov 20, 2023
2eaa9aa
use Key struct in state methods
samliok Nov 20, 2023
2526ced
comments
samliok Nov 20, 2023
f457e23
key as ref and lint
samliok Nov 20, 2023
a5c652c
update sdk macro naming
samliok Nov 20, 2023
baa312a
forgot one is_context() method
samliok Nov 20, 2023
2158524
update public macro to convert from raw_ptr
samliok Nov 20, 2023
3e5b797
harden public macro, and converted params logic
samliok Nov 20, 2023
16d9dcd
reorder comments
samliok Nov 20, 2023
bfb7338
HostArgument type + Vec Example(rust)
samliok Nov 21, 2023
17cab87
added marshaling to go side, and token examples work(hype)
samliok Nov 21, 2023
a9993af
letsgoo, finally passing dynamic vecs working(needs clean up(from deb…
samliok Nov 21, 2023
b487e6a
cleanup logs
samliok Nov 21, 2023
a483cf5
combine HostArg + Arg
samliok Nov 21, 2023
1793386
vec<u8> -> &[u8]
samliok Nov 21, 2023
d97f278
avoid reallocation in as_bytes
samliok Nov 21, 2023
96190d4
temporary string example/method
samliok Nov 21, 2023
077d36b
remove VecArg junk
samliok Nov 21, 2023
ffa1dd9
cargo fmt
samliok Nov 21, 2023
21f1d81
lint
samliok Nov 21, 2023
67b69c4
flatten out tuple
samliok Nov 22, 2023
378d578
simplify ParamKind logic
samliok Nov 22, 2023
dd66f33
borsh serialization working(go/rust)
samliok Nov 22, 2023
6c45d95
move all from serde -> borsh(except simulator)
samliok Nov 22, 2023
033004a
call_program function
samliok Nov 22, 2023
4c15795
re added arg types, going to address call_program in another pr
samliok Nov 22, 2023
8dd98dc
unsafe + temp removal
samliok Nov 22, 2023
754c7bc
lint
samliok Nov 22, 2023
7c0ca41
go lint
samliok Nov 22, 2023
1e8fd38
mint recipient struct
samliok Nov 23, 2023
d97a849
mint recipient struct
samliok Nov 23, 2023
912ecc1
Update x/programs/rust/wasmlanche_sdk/src/types.rs
samliok Nov 23, 2023
0944839
lint
samliok Nov 23, 2023
c272666
remove temp go stuff
samliok Nov 24, 2023
813ffdc
tidy
samliok Nov 28, 2023
3d61aee
remove comment
samliok Nov 28, 2023
2f0090e
decompose newPtr method + nits
samliok Nov 30, 2023
ede907a
remove fixedLengthKey method
samliok Nov 30, 2023
bc9fa1a
refactor host calls
samliok Dec 1, 2023
968661c
return result in from_raw_ptr
samliok Dec 2, 2023
ef4cb9d
merge main
samliok Dec 2, 2023
a4f4a15
conflicts from merge
samliok Dec 2, 2023
236e963
represent pointers as *const u8
samliok Dec 2, 2023
2d240c8
comments
samliok Dec 2, 2023
cdd09a3
remove import
samliok Dec 2, 2023
4449ad7
remove len
samliok Dec 2, 2023
ed5ab6a
rust lint
samliok Dec 2, 2023
f207a5f
lfg some progress on serialize_param macro
samliok Dec 3, 2023
1ee07b1
why are we not putting units in second runtime?
samliok Dec 3, 2023
e1c319a
comments
samliok Dec 3, 2023
6c2cde5
counter test cleanup
samliok Dec 3, 2023
5d29b27
comment cleanup
samliok Dec 3, 2023
7db59e5
nits
samliok Dec 3, 2023
d19f883
program struct refactor
samliok Dec 4, 2023
ef3d0cc
yesss, external function calls working(needs cleanup)
samliok Dec 4, 2023
da30514
cleanup
samliok Dec 5, 2023
2f9a228
cargo fmt
samliok Dec 5, 2023
c879e1c
Fix token benchmark
hexfusion Dec 5, 2023
bc6741e
nits + Deref on Key struct
samliok Dec 6, 2023
b6facbc
bitwise operations on ptrs passing to the host
samliok Dec 8, 2023
cbe1000
Merge remote-tracking branch 'origin/refactor-statekey' into refactor…
samliok Dec 8, 2023
5ea23bb
bitwise operations working pasing parameters into wasm
samliok Dec 8, 2023
c4efc96
full circle -> external function calls(needs cleanup)
samliok Dec 8, 2023
caab4f3
ptr cleanup
samliok Dec 8, 2023
5595b01
comments and nit go cleanup
samliok Dec 8, 2023
245b06c
updated call_program host function
samliok Dec 9, 2023
764077d
SmartPtr type & cleanup utils
samliok Dec 10, 2023
cc96b79
go lint
samliok Dec 10, 2023
a17803d
rust lint
samliok Dec 10, 2023
158bfd2
update program impl
samliok Dec 10, 2023
2bd46d4
return result from rust side host function wrappers
samliok Dec 10, 2023
ee4507d
small rename
samliok Dec 10, 2023
2ab2bb1
rust lint
samliok Dec 10, 2023
f991a95
into_bytes function
samliok Dec 11, 2023
866febf
smart ptr method refactor
samliok Dec 11, 2023
f2e9f72
comments
samliok Dec 11, 2023
7cb99c4
int64 conversionn
samliok Dec 11, 2023
40f45d0
Merge branch 'main' into refactor-statekey
hexfusion Dec 12, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 22 additions & 14 deletions x/programs/examples/counter_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ var counterProgramBytes []byte
func TestCounterProgram(t *testing.T) {
require := require.New(t)
db := newTestDB()
maxUnits := uint64(40000)
cfg, err := runtime.NewConfigBuilder().Build()
maxUnits := uint64(80000000)
cfg, err := runtime.NewConfigBuilder().WithDebugMode(true).Build()
require.NoError(err)

ctx, cancel := context.WithCancel(context.Background())
Expand All @@ -52,7 +52,7 @@ func TestCounterProgram(t *testing.T) {
err = storage.SetProgram(ctx, db, programID, counterProgramBytes)
require.NoError(err)

programIDPtr, err := runtime.WriteBytes(rt.Memory(), programID[:])
programIDPtr, err := newParameterPtr(ctx, programID, rt)
require.NoError(err)

// generate alice keys
Expand All @@ -77,10 +77,11 @@ func TestCounterProgram(t *testing.T) {
// meter.
rt2 := runtime.New(log, cfg, supported.Imports())
err = rt2.Initialize(ctx, counterProgramBytes, runtime.NoUnits)

require.NoError(err)

// define max units to transfer to second runtime
unitsTransfer := uint64(10000)
unitsTransfer := uint64(150000)

// transfer the units from the original runtime to the new runtime before
// any calls are made.
Expand All @@ -92,7 +93,7 @@ func TestCounterProgram(t *testing.T) {
err = storage.SetProgram(ctx, db, program2ID, counterProgramBytes)
require.NoError(err)

programID2Ptr, err := runtime.WriteBytes(rt2.Memory(), program2ID[:])
programID2Ptr, err := newParameterPtr(ctx, program2ID, rt2)
require.NoError(err)

// write alice's key to stack and get pointer
Expand All @@ -105,14 +106,16 @@ func TestCounterProgram(t *testing.T) {
require.Equal(int64(1), result[0])

// increment alice's counter on program 2 by 10
result, err = rt2.Call(ctx, "inc", programID2Ptr, alicePtr2, 10)
incAmount := int64(10)
incAmountPtr, err := newParameterPtr(ctx, incAmount, rt2)
samliok marked this conversation as resolved.
Show resolved Hide resolved
require.NoError(err)
result, err = rt2.Call(ctx, "inc", programID2Ptr, alicePtr2, incAmountPtr)
require.NoError(err)
require.Equal(int64(1), result[0])

result, err = rt2.Call(ctx, "get_value", programID2Ptr, alicePtr2)
require.NoError(err)
require.Equal(int64(10), result[0])

require.Equal(incAmount, result[0])
// stop the runtime to prevent further execution
rt2.Stop()

Expand All @@ -125,7 +128,9 @@ func TestCounterProgram(t *testing.T) {
}

// increment alice's counter on program 1
result, err = rt.Call(ctx, "inc", programIDPtr, alicePtr, 1)
onePtr, err := newParameterPtr(ctx, int64(1), rt)
samliok marked this conversation as resolved.
Show resolved Hide resolved
require.NoError(err)
result, err = rt.Call(ctx, "inc", programIDPtr, alicePtr, onePtr)
require.NoError(err)
require.Equal(int64(1), result[0])

Expand All @@ -137,22 +142,25 @@ func TestCounterProgram(t *testing.T) {
)

// write program id 2 to stack of program 1
programID2Ptr, err = runtime.WriteBytes(rt.Memory(), program2ID[:])
programID2Ptr, err = newParameterPtr(ctx, program2ID, rt)
require.NoError(err)

caller := programIDPtr
target := programID2Ptr
maxUnitsProgramToProgram := int64(10000)
maxUnitsProgramToProgram := int64(100000)
maxUnitsProgramToProgramPtr, err := newParameterPtr(ctx, maxUnitsProgramToProgram, rt)
require.NoError(err)

// increment alice's counter on program 2
result, err = rt.Call(ctx, "inc_external", caller, target, maxUnitsProgramToProgram, alicePtr, 5)
fivePtr, err := newParameterPtr(ctx, int64(5), rt)
require.NoError(err)
result, err = rt.Call(ctx, "inc_external", caller, target, maxUnitsProgramToProgramPtr, alicePtr, fivePtr)
require.NoError(err)
require.Equal(int64(1), result[0])

// expect alice's counter on program 2 to be 15
result, err = rt.Call(ctx, "get_value_external", caller, target, maxUnitsProgramToProgram, alicePtr)
result, err = rt.Call(ctx, "get_value_external", caller, target, maxUnitsProgramToProgramPtr, alicePtr)
require.NoError(err)
require.Equal(int64(15), result[0])

require.Greater(rt.Meter().GetBalance(), uint64(0))
}
73 changes: 36 additions & 37 deletions x/programs/examples/imports/program/program.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package program

import (
"context"
"encoding/binary"
"fmt"

"go.uber.org/zap"
Expand All @@ -13,8 +14,9 @@ import (
"github.com/ava-labs/avalanchego/utils/logging"
"github.com/bytecodealliance/wasmtime-go/v14"

"github.com/ava-labs/hypersdk/codec"
"github.com/ava-labs/hypersdk/consts"
"github.com/ava-labs/hypersdk/state"
"github.com/ava-labs/hypersdk/x/programs/examples/imports"
"github.com/ava-labs/hypersdk/x/programs/examples/storage"
"github.com/ava-labs/hypersdk/x/programs/runtime"
)
Expand Down Expand Up @@ -60,8 +62,8 @@ func (i *Import) Register(link runtime.Link, meter runtime.Meter, imports runtim
// callProgramFn makes a call to an entry function of a program in the context of another program's ID.
func (i *Import) callProgramFn(
caller *wasmtime.Caller,
callerIDPtr int64,
programIDPtr int64,
callerIDPtr int32,
samliok marked this conversation as resolved.
Show resolved Hide resolved
programIDPtr int32,
maxUnits int64,
functionPtr,
functionLen,
Expand All @@ -71,7 +73,6 @@ func (i *Import) callProgramFn(
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
memory := runtime.NewMemory(runtime.NewExportClient(caller))

// get the entry function for invoke to call.
functionBytes, err := memory.Range(uint64(functionPtr), uint64(functionLen))
if err != nil {
Expand Down Expand Up @@ -132,25 +133,15 @@ func (i *Import) callProgramFn(
}
}()

// write the program id to the new runtime memory
ptr, err := runtime.WriteBytes(rt.Memory(), programIDBytes)
if err != nil {
i.log.Error("failed to write program id to memory",
zap.Error(err),
)
return -1
}

argsBytes, err := memory.Range(uint64(argsPtr), uint64(argsLen))
if err != nil {
i.log.Error("failed to read program args name from memory",
zap.Error(err),
)
return -1
}

// sync args to new runtime and return arguments to the invoke call
params, err := getCallArgs(ctx, rt.Memory(), argsBytes, ptr)
params, err := getCallArgs(ctx, rt.Memory(), argsBytes, programIDBytes)
if err != nil {
i.log.Error("failed to unmarshal call arguments",
zap.Error(err),
Expand All @@ -170,31 +161,39 @@ func (i *Import) callProgramFn(
return int64(res[0])
}

func getCallArgs(ctx context.Context, memory runtime.Memory, buffer []byte, invokeProgramID int64) ([]int64, error) {
func getCallArgs(ctx context.Context, memory runtime.Memory, buffer []byte, programIDBytes []byte) ([]int64, error) {
// first arg contains id of program to call
args := []int64{invokeProgramID}
p := codec.NewReader(buffer, len(buffer))
i := 0
for !p.Empty() {
size := p.UnpackInt64(true)
isInt := p.UnpackBool()
if isInt {
valueInt := p.UnpackInt64(true)
args = append(args, valueInt)
} else {
valueBytes := make([]byte, size)
p.UnpackFixedBytes(int(size), &valueBytes)
ptr, err := runtime.WriteBytes(memory, valueBytes)
if err != nil {
return nil, err
}
args = append(args, ptr)
}
i++
invokeProgramIDPtr, err := runtime.WriteBytes(memory, programIDBytes)
if err != nil {
return nil, err
}
if p.Err() != nil {
return nil, fmt.Errorf("failed to unpack arguments: %w", p.Err())
argPtr, err := imports.ToPtrArgument(invokeProgramIDPtr, uint32(len(programIDBytes)))
if err != nil {
return nil, err
}

args := []int64{argPtr}

for i := 0; i < len(buffer); {
// unpacks uint32
lenBytes := buffer[i : i+consts.Uint32Len]
length := binary.BigEndian.Uint32(lenBytes)

valueBytes := buffer[i + consts.Uint32Len : i+ consts.Uint32Len + int(length)]
i += int(length) + consts.Uint32Len

// every argument is a pointer
ptr, err := runtime.WriteBytes(memory, valueBytes)
samliok marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return nil, err
}
argPtr, err := imports.ToPtrArgument(ptr, length)
if err != nil {
return nil, err
}
args = append(args, argPtr)
}

return args, nil
}

Expand Down
76 changes: 25 additions & 51 deletions x/programs/examples/imports/pstate/pstate.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ import (
"go.uber.org/zap"

"github.com/ava-labs/avalanchego/database"
"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/utils/logging"

"github.com/ava-labs/hypersdk/state"
"github.com/ava-labs/hypersdk/x/programs/examples/imports"
"github.com/ava-labs/hypersdk/x/programs/examples/storage"
"github.com/ava-labs/hypersdk/x/programs/runtime"
)
Expand Down Expand Up @@ -55,32 +55,31 @@ func (i *Import) Register(link runtime.Link, meter runtime.Meter, _ runtime.Supp
if err := link.FuncWrap(Name, "get", i.getFn); err != nil {
return err
}
if err := link.FuncWrap(Name, "len", i.getLenFn); err != nil {
return err
}
samliok marked this conversation as resolved.
Show resolved Hide resolved

return nil
}

func (i *Import) putFn(caller *wasmtime.Caller, idPtr int64, keyPtr int32, keyLength int32, valuePtr int32, valueLength int32) int32 {
memory := runtime.NewMemory(runtime.NewExportClient(caller))
programIDBytes, err := memory.Range(uint64(idPtr), uint64(ids.IDLen))
func (i *Import) putFn(caller *wasmtime.Caller, id int64, key int64, value int64) int32 {
client := runtime.NewExportClient(caller)
// memory := runtime.NewMemory(client)
programIDBytes, err := imports.GetBytesFromArgPtr(client, id)
if err != nil {
i.log.Error("failed to read program id from memory",
zap.Error(err),
)
return -1
}

keyBytes, err := memory.Range(uint64(keyPtr), uint64(keyLength))
keyBytes, err := imports.GetBytesFromArgPtr(client, key)
if err != nil {
i.log.Error("failed to read key from memory",
zap.Error(err),
)
return -1
}
zap.Error(err),
)
return -1
}

valueBytes, err := imports.GetBytesFromArgPtr(client, value)

valueBytes, err := memory.Range(uint64(valuePtr), uint64(valueLength))
if err != nil {
i.log.Error("failed to read value from memory",
zap.Error(err),
Expand All @@ -100,24 +99,24 @@ func (i *Import) putFn(caller *wasmtime.Caller, idPtr int64, keyPtr int32, keyLe
return 0
}

func (i *Import) getLenFn(caller *wasmtime.Caller, idPtr int64, keyPtr int32, keyLength int32) int32 {
memory := runtime.NewMemory(runtime.NewExportClient(caller))
programIDBytes, err := memory.Range(uint64(idPtr), uint64(ids.IDLen))
func (i *Import) getFn(caller *wasmtime.Caller, id int64, key int64) int64 {
client := runtime.NewExportClient(caller)
memory := runtime.NewMemory(client)
programIDBytes, err := imports.GetBytesFromArgPtr(client, id)
if err != nil {
i.log.Error("failed to read program id from memory",
zap.Error(err),
)
return -1
}

keyBytes, err := memory.Range(uint64(keyPtr), uint64(keyLength))
keyBytes, err := imports.GetBytesFromArgPtr(client, key)
if err != nil {
i.log.Error("failed to read key from memory",
zap.Error(err),
)
return -1
}

k := storage.ProgramPrefixKey(programIDBytes, keyBytes)
val, err := i.mu.GetValue(context.Background(), k)
if err != nil {
Expand All @@ -128,55 +127,30 @@ func (i *Import) getLenFn(caller *wasmtime.Caller, idPtr int64, keyPtr int32, ke
}
return -1
}

return int32(len(val))
}

func (i *Import) getFn(caller *wasmtime.Caller, idPtr int64, keyPtr int32, keyLength int32, valLength int32) int32 {
memory := runtime.NewMemory(runtime.NewExportClient(caller))
programIDBytes, err := memory.Range(uint64(idPtr), uint64(ids.IDLen))
if err != nil {
i.log.Error("failed to read program id from memory",
zap.Error(err),
)
return -1
}

keyBytes, err := memory.Range(uint64(keyPtr), uint64(keyLength))
if err != nil {
i.log.Error("failed to read key from memory",
i.log.Error("failed to convert program id to id",
zap.Error(err),
)
return -1
}

k := storage.ProgramPrefixKey(programIDBytes, keyBytes)
val, err := i.mu.GetValue(context.Background(), k)
// prepend the length so that the program can grab the correct number of bytes
ptr, err := runtime.WriteBytes(memory, val)
if err != nil {
if !errors.Is(err, database.ErrNotFound) {
i.log.Error("failed to get value from storage",
{
i.log.Error("failed to write to memory",
zap.Error(err),
)
}
return -1
}

argPtr, err := imports.ToPtrArgument(ptr, uint32(len(val)))
if err != nil {
i.log.Error("failed to convert program id to id",
i.log.Error("failed to convert ptr to argument",
zap.Error(err),
)
return -1
}

ptr, err := runtime.WriteBytes(memory, val)
if err != nil {
{
i.log.Error("failed to write to memory",
zap.Error(err),
)
}
return -1
}

return int32(ptr)
return argPtr
}
Loading
Loading