-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add experimental daemon plugin for 9P2000.L support #6612
Closed
Closed
Changes from all commits
Commits
Show all changes
102 commits
Select commit
Hold shift + click to select a range
2a02a91
plugin/fs: initial experiment
djdv ef75786
plugin/fs: Basic Readdir
djdv 43a38b6
plugin/fs: Basic ReadAt
djdv 7a80532
plugin/fs: Fix a bunch of path related issues
djdv 2d99302
plugin/fs: Fix a bunch of metadata related issues
djdv 75618a5
plugin/fs: linting
djdv 07534a3
plugin/fs: use config
djdv 0d28489
plugin/fs: Better init/config
djdv 7bb358f
plugin/fs: Betterer init/config
djdv 6a17332
plugin/fs: consistent protocol identifier
djdv 090f7cd
plugin/fs: text secession
djdv 19d5332
squashme-deps
djdv 353007e
plugin/fs: client fixes
djdv 6e7ea19
plugin/fs: add PinFS test
djdv 28c5cff
plugin/fs: add root test + change pin test
djdv 81b9a06
IPFS-test WIP
djdv 864dbfe
deps
djdv 35eee21
fix fs test on linux
djdv 483845c
CR feedback + docs WIP, linting, fix freebsd builds
djdv 3876302
openbsd builds
djdv d9fd656
fix doc format
djdv f48f4a8
stop tracking client-lib
djdv 4bb5712
docfix
djdv 760d3c6
wait for fs service to return on close + remove listener on Windows i…
djdv 6e87be8
change how IPFS reads directory streams
djdv 7ff0d1a
fix readdir offset + add test
djdv 99df410
fix: use environment variable when set
djdv dc6287d
docs: wrong port + add more to the client example
djdv b8c4a3c
markdown fixes
djdv 0997454
the linebreaks too
djdv 3ee383a
doc: add a table of the root index mapping
djdv 15e894a
CR bundle (split later)
djdv e5ec672
qid-fix
djdv c653fc5
clone-fix
djdv b6d1506
change config handling (needs tests)
djdv fbf5bca
delay listener init
djdv 888c33d
use multicodec standard
djdv 3017586
don't implicitly timestamp core objects
djdv 6507b34
use a standard blocksize
djdv 4a471d6
stop listening sooner
djdv afd9cb1
ipfs read fixes
djdv e8c1a5e
docs: experimental stability before feature reality
djdv a2a9e8b
readdir improvments
djdv 0a6110d
readdir end of stream fix
djdv 1dd3125
store ipfs reference on open for use in read
djdv 7b10b5f
better constrain/contextualize operation calls
djdv 82780a1
change root construction
djdv 4c9a790
better walk logic (and more; todo:split commit)
djdv fb20a44
ipns initial
djdv 5ad516c
log linting
djdv 02f69b8
walk logic - this is better but still broken
djdv eaff0c9
misc cleanup and context changes
djdv 3ab6029
context consistency
djdv b6dee76
less broken stat
djdv 770bbce
small lint
djdv 4717f73
Revert "change config handling (needs tests)"
djdv 21b2d2c
rework config, remove windows manet hacks, remove ipns
djdv 62560e4
fix read offset and return values
djdv 11c5add
change mount docs, context WIP
djdv dd3a894
walk refactor
djdv 1636e1b
unbreak tests
djdv 63f6d27
possible fix for re-attachment
djdv 97f36a9
socket path template nit
djdv 3c56f5f
change attacher-constructors and walk clone-semantics
djdv 4c1ad5d
add basic root tests
djdv e3db71a
remove runtime check, already static checked
djdv b2a43b8
fix attach for other roots too
djdv 9cc06ae
traversal/construction refactor
djdv e79e371
ipns refactor WIP
djdv d1a1170
change/fix node derive logic
djdv be83005
update experimental doc
djdv fbac898
linting
djdv 62cf47e
test: don't compare directory size
djdv e8ccc6b
list ipns in the root
djdv 6d92612
context refactor wip
djdv 294c3ea
actually cancel the context + lint
djdv 8ce5e51
actually use the context too
djdv 8328485
don't readdir forever during Linux tests
djdv db0bcba
don't lose err value
djdv 7bb94a4
readdir fixes
djdv e029c7d
gomod deps
djdv 8cee086
changes and tests around plugin close
djdv 2a495c7
lol unsaved buffers
djdv 161d86a
expose WalkRef and generalize Backtrack
djdv 9ef8e6a
extract WalkRef and cleanup related use
djdv 545d666
unexport NinePath
djdv 9024b95
inheritance changes related to the base clases
djdv a0465f4
small lint and docs
djdv 2aead06
change attach clone method
djdv ec9f8ea
unbreak root attrs
djdv c744a00
cleanup attachers
djdv 65b677f
slightly saner test strings
djdv 7ce0e9b
test open
djdv 4c6b8ee
allow ipfs root to be read
djdv 6e654db
fix context for proxy roots
djdv 3501592
fix
djdv 07db2b9
doc pass WIP
djdv 984f155
make sure test defers gets called
djdv 0fa5983
try not to leak
djdv c9faaa7
change Plugin Close procedure
djdv c7cfd71
change Plugin Close procedure part 2
djdv c294ac8
don't swallow Read return value
djdv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| /* | ||
| Package filesystem is an experimental package that implements the go-ipfs daemon plugin interface | ||
| and defines the plugin's config structure. The plugin itself exposes file system services over a multiaddr listener. | ||
|
|
||
| By default, we try to expose the IPFS namespace using the 9P2000.L protocol, over a unix domain socket | ||
| (located at $IPFS_PATH/filesystem.9P.sock) | ||
|
|
||
| To set the multiaddr listen address, you may use the environment variable $IPFS_FS_ADDR, or set the option in the node's config file | ||
| via `ipfs config --json 'Plugins.Plugins.filesystem.Config "Config":{"Service":{"9P":"/ip4/127.0.0.1/tcp/564"}}'` | ||
| To disable this plugin entirely, use: `ipfs config --json Plugins.Plugins.filesystem.Disabled true` | ||
| */ | ||
| package filesystem | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| package filesystem | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "fmt" | ||
| "net" | ||
| "os" | ||
| "path/filepath" | ||
| "strings" | ||
|
|
||
| "github.com/hugelgupf/p9/p9" | ||
| plugin "github.com/ipfs/go-ipfs/plugin" | ||
| fsnodes "github.com/ipfs/go-ipfs/plugin/plugins/filesystem/nodes" | ||
| nodeopts "github.com/ipfs/go-ipfs/plugin/plugins/filesystem/nodes/options" | ||
| logging "github.com/ipfs/go-log" | ||
| coreiface "github.com/ipfs/interface-go-ipfs-core" | ||
| "github.com/mitchellh/mapstructure" | ||
| "github.com/multiformats/go-multiaddr" | ||
| manet "github.com/multiformats/go-multiaddr-net" | ||
| ) | ||
|
|
||
| var ( | ||
| _ plugin.PluginDaemon = (*FileSystemPlugin)(nil) // impl check | ||
|
|
||
| // Plugins is an exported list of plugins that will be loaded by go-ipfs. | ||
| Plugins = []plugin.Plugin{ | ||
| &FileSystemPlugin{}, //TODO: individually name implementations: &P9{} | ||
| } | ||
|
|
||
| logger logging.EventLogger | ||
| ) | ||
|
|
||
| func init() { | ||
| logger = logging.Logger("plugin/filesystem") | ||
| } | ||
|
|
||
| type FileSystemPlugin struct { | ||
| ctx context.Context | ||
| cancel context.CancelFunc | ||
|
|
||
| addr multiaddr.Multiaddr | ||
| listener manet.Listener | ||
| closed chan struct{} | ||
| serverErr error | ||
| } | ||
|
|
||
| func (*FileSystemPlugin) Name() string { | ||
| return PluginName | ||
| } | ||
|
|
||
| func (*FileSystemPlugin) Version() string { | ||
| return PluginVersion | ||
| } | ||
|
|
||
| func (fs *FileSystemPlugin) Init(env *plugin.Environment) error { | ||
djdv marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| logger.Info("Initializing 9P resource server...") | ||
| if fs.addr != nil { | ||
| return fmt.Errorf("already initialized with %s", fs.addr.String()) | ||
| } | ||
|
|
||
| // stabilise repo path; our template depends on this | ||
| if !filepath.IsAbs(env.Repo) { | ||
| absRepo, err := filepath.Abs(env.Repo) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| env.Repo = absRepo | ||
| } | ||
|
|
||
| cfg := &Config{} | ||
| // load config from file or initialize it | ||
| if env.Config != nil { | ||
| if err := mapstructure.Decode(env.Config, cfg); err != nil { | ||
| return err | ||
| } | ||
| } else { | ||
| cfg = defaultConfig() | ||
| } | ||
|
|
||
| var addrString string | ||
| // allow environment variable to override config values | ||
| if envAddr := os.ExpandEnv(EnvAddr); envAddr != "" { | ||
| addrString = EnvAddr | ||
| } else { | ||
| addrString = cfg.Service[defaultService] | ||
| } | ||
|
|
||
| // expand string templates and initialize listening addr | ||
| templateRepoPath := env.Repo | ||
| if strings.HasPrefix(addrString, "/unix") { | ||
| // prevent template from expanding to double slashed paths like `/unix//home/...` | ||
| // but allow it to expand to `/unix/C:\Users...` | ||
| templateRepoPath = strings.TrimPrefix(templateRepoPath, "/") | ||
| } | ||
|
|
||
| addrString = os.Expand(addrString, configVarMapper(templateRepoPath)) | ||
|
|
||
| ma, err := multiaddr.NewMultiaddr(addrString) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| fs.addr = ma | ||
|
|
||
| logger.Info("9P resource server okay for launch") | ||
| return nil | ||
| } | ||
|
|
||
| func (fs *FileSystemPlugin) Start(core coreiface.CoreAPI) error { | ||
| logger.Info("Starting 9P resource server...") | ||
| if fs.addr == nil { | ||
| return fmt.Errorf("Start called before plugin Init") | ||
| } | ||
|
|
||
| // make sure we're not in use already | ||
| if fs.listener != nil { | ||
| return fmt.Errorf("already started and listening on %s", fs.listener.Addr()) | ||
| } | ||
|
|
||
| // make sure sockets are not in use already (if we're using them) | ||
| err := removeUnixSockets(fs.addr) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| // launch the listener | ||
| listener, err := manet.Listen(fs.addr) | ||
| if err != nil { | ||
| logger.Errorf("9P listen error: %s\n", err) | ||
| return err | ||
| } | ||
| fs.listener = listener | ||
|
|
||
| // construct and launch the 9P resource server | ||
| fs.ctx, fs.cancel = context.WithCancel(context.Background()) | ||
| fs.closed = make(chan struct{}) | ||
|
|
||
| opts := []nodeopts.AttachOption{nodeopts.Logger(logging.Logger("9root"))} | ||
| server := p9.NewServer(fsnodes.RootAttacher(fs.ctx, core, opts...)) | ||
| go func() { | ||
| // run the server until the listener closes | ||
| // store error on the fs object then close our syncing channel (see use in `Close` below) | ||
|
|
||
| err := server.Serve(manet.NetListener(fs.listener)) | ||
|
|
||
| // [async] we expect `net.Accept` to fail when the filesystem has been canceled | ||
| if fs.ctx.Err() != nil { | ||
| // non-'accept' ops are not expected to fail, so their error is preserved | ||
| var opErr *net.OpError | ||
| if errors.As(fs.serverErr, &opErr) && opErr.Op != "accept" { | ||
| fs.serverErr = err | ||
| } | ||
| } else { | ||
| // unexpected failure during operation | ||
| fs.serverErr = err | ||
| } | ||
|
|
||
| close(fs.closed) | ||
| }() | ||
|
|
||
| logger.Infof("9P service is listening on %s\n", fs.listener.Addr()) | ||
| return nil | ||
| } | ||
|
|
||
| func (fs *FileSystemPlugin) Close() error { | ||
Stebalien marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| logger.Info("9P server requested to close") | ||
| if fs.addr == nil { // forbidden | ||
| return fmt.Errorf("Close called before plugin Init") | ||
| } | ||
|
|
||
| // synchronization between plugin interface <-> fs server | ||
| if fs.closed != nil { // implies `Start` was called prior | ||
| fs.cancel() // stop and prevent all fs operations, signifies "closing" intent | ||
| fs.listener.Close() // stop accepting new clients | ||
| <-fs.closed // wait for the server thread to set the error value | ||
| fs.listener = nil // reset `Start` conditions | ||
| fs.closed = nil | ||
| } | ||
| // otherwise we were never started to begin with; default/initial value will be returned | ||
| return fs.serverErr | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling this returns
Error: failed to get config value: " key has no attributes"when using the default config.Is there some flag we can pass to force creation of keys that don't exist? If not should we add one?
Otherwise, we need more instructing for the user. As it stands they'd need to manually add the Plugin section + our plugin's config within it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not yet. Really, we should change how we do config type checking (see https://github.com/ipfs/go-ipfs/blob/7e7b76259f655c5aa321e622a38619587a5a02a8/repo/fsrepo/fsrepo.go#L589). Basically, instead of manually type checking like that, we should just set the key and see if it serializes correctly.
The tricky part will be making sure the error doesn't suck.