Add 'ipfs file ls …'

[wking]

This is mostly to get an interface for listing the file-sizes for Unix
files (vs. the Merkle-ancestor sizes listed by ipfs ls …). For
background, see #543. For details on the individual commits, see the
commit messages.

The intended space for ipfs unixfs … is for folks operating with
Unix-filesystem trees (obviously), so the API should tranparently
resolve any fanout. The current Merkle-level commands (ipfs ls …,
etc.) on the other hand should be operating below the UnixFS level.
They probably shouldn't handle fanout, and they almost certainly
shouldn't be converting Merkle nodes to UnixFS nodes. For example,
ipfs cat … might print the appended set of all constituent chunks,
but it shouldn't be stripping the magic Data wrappers (Type,
Filesize, Blocksizes, … in unixfs_pb.Data), because ‘ipfs cat …’
might be used to access non-UnixFS nodes.

I think there is space between the really-low-level ipfs object …
API and the UnixFS API we're starting here for commands that only add
fanout and typing to the base Merkle objects. Then the UnixFS
implementation can build on that. For example, the ipfs unixfs ls …
implementation here doesn't currently handle fanned-out directories.
Blocking issues for building such an intermediate API include generic
typing (vs. the Protobuf magic in unixfs_pb.Data, see ipfs/ipfs#36)
and generic fanout (see ipfs/specs#10).

There are a few consistency issues with the exising ipfs ls … that I
point out in the commit messages, and we'll want to resolve those
before merging this PR.

[jbenet]

@wking let's use ipfs files as the command. ipfs unixfs will be weird to most users.

[jbenet]

I think there is space between the really-low-level ipfs object …
API and the UnixFS API we're starting here for commands that only add
fanout and typing to the base Merkle objects.

Agree. not sure what this would look like yet.

[wking]

On Tue, Jun 09, 2015 at 04:08:14PM -0700, Juan Batiz-Benet wrote:

let's use ipfs files as the command? ipfs unixfs will be weird
to most users.

Ok. How much of the code should be renamed from unixfs → files?
core/commands/files? The current unixfs/ (protobuf definitions,
etc.)? None of it?

[whyrusleeping]

just this command should be files. unixfs is fine in the codebase

[jbenet]

just this command should be files. unixfs is fine in the codebase

exactly.

we use unixfs, but expose ipfs files in the commandline because the cli would be silly otherwise. it could be put in a unixfs standalone tool. (like ipfs name could go into ipns)

[GitCop]

There were the following issues with your Pull Request

• Commit: 7c91e3a
  • Invalid signoff. Commit message must end with License: MIT
    Signed-off-by: .* <.*>

Guidelines are available at: https://github.com/ipfs/community/blob/master/docs/commit-message.md Your feedback on GitCop is welcome on the following issue: ipfs/infra#23

---

This message was auto-generated by https://gitcop.com

[wking]

@jbenet: can you weigh in on the following 'ipfs ls ...' vs. 'ipfs
file ls ...' consistency issues (I pointed to the commits in my
initial post, but since there haven't been comments on them I thought
I'd call them out again):

• name markup (e.g. "bin" -> "bin/") vs. explicit type name
  (e.g. "Directory"). See d0ce146
• dropped minute-timeout context.
  bdefbbb
• dropped trailing blank line.
  bdefbbb
• Hash vs. Argument in LsObjects (shows up in JSON output). See
  30f7819
• string types in LsLink (shows up in JSON output).
  4232e9c

[jbenet]

name markup (e.g. "bin" -> "bin/") vs. explicit type name
(e.g. "Directory"). See d0ce146

Oh interesting. Let me think about it.

• FWIW, ls is very clear: it lists <dir>/ for directories.
• There's also the d in mode, so it can list the type in some way.

Maybe we should have both? so that ipfs files ls dir -q shows:

file1
file2
dir3/
dir4/
file5

and is compatible with tools that are ls compatible?

One caveat: does this require downloading the children too? that's usually expensive and worth noting.

• dropped minute-timeout context.
  bdefbbb

I think this is right?

@whyrusleeping pls take a look.

• dropped trailing blank line.
  bdefbbb

dropping a blank line is fine, but we should still have nice CLI output, i.e. it should end with a newline (unless -q or a similar flag demands otherwise)

• Hash vs. Argument in LsObjects (shows up in JSON output). See
  30f7819

what is Argument? I don't follow. I need to look at this PR in depth.

• string types in LsLink (shows up in JSON output).
  4232e9c

I think we need universal self-description here. basically IPFS-LD. We'll regret it later otherwise. Let me think about this one more.

[wking]

On Thu, Jun 11, 2015 at 08:14:35PM -0700, Juan Batiz-Benet wrote:

Maybe we should have both? so that ipfs files ls dir -q … is
compatible with tools that are ls compatible?

In POSIX, the -q option is about hiding control characters 1. Maybe
you meant -F or -p?

One caveat: does this require downloading the children too? that's
usually expensive and worth noting.

Yes, although you'll only need to download the first Merkle object to
figure out it's type (e.g. not the entirety of a chunked file or
fanned-out directory).

• dropped trailing blank line.
  bdefbbb

… i.e. it should end with a newline (unless -q or a similar flag
demands otherwise)

For non-JSON output this is how it was, is, and ever shall be. This
just drops the trailing blank line (e.g. \n\n → \n). I'm happy to
have our default JSON encoder spit out a trailing newline, but I
didn't see an easy way to do that (discussion in 30f7819).

• Hash vs. Argument in LsObjects (shows up in JSON output). See
  30f7819

what is Argument? I don't follow. I need to look at this PR in
depth.

The command-line argument for which you're getting a result block.
For example:

$ ipfs file ls
:

File

Somewhat similarly:

$ ls -l bin/ .bashrc
lrwxrwxrwx 1 wking wking 51 Nov 25 2013 .bashrc -> /home/wking/src/dotfiles/public/patched-src/.bashrc

bin/:
total 21824
lrwxrwxrwx 1 wking wking 37 Nov 25 2013 bin/dotfiles.sh -> ../src/dotfiles-framework/dotfiles.sh
-rwxr-xr-x 1 wking wking 22347064 Jun 11 21:02 ipfs

[jbenet]

In POSIX, the -q option is about hiding control characters [1]. Maybe
you meant -F or -p?

No, i meant -q. In many git commands and in in many ipfs commands -q means quiet and removes unnecessary output (like a trailing newline).

Yes, although you'll only need to download the first Merkle object to
figure out it's type (e.g. not the entirety of a chunked file or
fanned-out directory).

still more expensive. but ok.

I'm happy to have our default JSON encoder spit out a trailing newline

we can just add it in the relevant commands

The command-line argument for which you're getting a result block.
For example:

ls only includes the output when there's multiple to differentiate between. If that's the case here, great.

It's getting difficult to follow the formats in the (convoluted) cmdslib marshaller code. the sharness tests are better.

perhaps one thing we can do is start compiling the cli interface design into per-command docs we derive the implementations + the sharness tests from. (ideally, we could just feed that to a "spec2sharness" tool that would check each example case or something. May be an upgrade to how sharness does things.

[wking]

On Thu, Jun 11, 2015 at 09:36:05PM -0700, Juan Batiz-Benet wrote:

In POSIX, the -q option is about hiding control characters 1. Maybe
you meant -F or -p?

No, i meant -q. In many git commands and in in many ipfs commands
-q means quiet and removes unnecessary output (like a trailing
newline).

Removing trailing newlines surprises me. Can you point me to the Git
command that interprets -q in that way?

I'm happy to have our default JSON encoder spit out a trailing
newline

we can just add it in the relevant commands

I'd expect “the relevant commands” is “anything called from the
command-line”.

The command-line argument for which you're getting a result block.
For example:

ls only includes the output when there's multiple to
differentiate between. If that's the case here, great.

Yup 1. The changes in 30f7819 just make sure we always set it in
the JSON output (even though it's not always set in the text output).

perhaps one thing we can do is start compiling the cli interface
design into per-command docs we derive the implementations + the
sharness tests from. (ideally, we could just feed that to a
"spec2sharness" tool that would check each example case or
something. May be an upgrade to how sharness does things.

That sounds nice, but I'd rather not fall down the ipfs/specs#12
rabbit hole (machine-parsable specs for interactive systems are hard
;). In the absence of such a command already existing, I think the
best bet is to just iterate on the sharness tests first, and then
leave it to the implementater (who is hopefully involved in the test
discussion) to get the tests to pass.

[jbenet]

@wking i think this is all you need

suppose i call

ls Foohash/bar Barhash

// i think this is all you strictly need
{
  "QmXvEg83dD5PvaDUqHV17iXXtTpZMnPrpKR2cGyU3oCK5": {
    "Links": [
      {
        "Name": "gentoo",
        "Hash": "QmXAuRiZKYpCvUmFFBBEzAMvvAgw9Aoxcf24DYaWnG43Er",
        "Size": 38775183,
        "Type": 1
      },
      {
        "Name": "protofs.md",
        "Hash": "QmWpn9xUDNNj3XvY4ixMgb8imiDZmGDr1qQLr1DBv8WAnG",
        "Size": 4953,
        "Type": 2
      }
    ]
  }
}

// to make life easier, can:
{
  "arguments": {
    "/ipfs/Fooohash/bar": "QmXvEg83dD5PvaDUqHV17iXXtTpZMnPrpKR2cGyU3oCK5",
    "/ipfs/Barhash": "QmXvEg83dD5PvaDUqHV17iXXtTpZMnPrpKR2cGyU3oCK5",
  },
  "objects": {
    "QmXvEg83dD5PvaDUqHV17iXXtTpZMnPrpKR2cGyU3oCK5": {
      "Links": [
        {
          "Name": "gentoo",
          "Hash": "QmXAuRiZKYpCvUmFFBBEzAMvvAgw9Aoxcf24DYaWnG43Er",
          "Size": 38775183,
          "Type": 1
        },
        {
          "Name": "protofs.md",
          "Hash": "QmWpn9xUDNNj3XvY4ixMgb8imiDZmGDr1qQLr1DBv8WAnG",
          "Size": 4953,
          "Type": 2
        }
      ]
    }
  }
}

[wking]

Rerolled to use Juan's new suggested JSON output.

[jbenet]

should check if the object is there. it may not be, and cause a crash below.

object, ok := output.Objects[hash]
if !ok { return error } 

[wking]

On Tue, Jun 16, 2015 at 03:51:31PM -0700, Juan Batiz-Benet wrote:

•           hash := output.Arguments[argument]
  

•           object := output.Objects[hash]
  

should check if the object is there. it may not be, and cause a
crash below.

Did you actually see a crash? We should be setting Objects[hash] for
every resolved argument 1. That said, I guess it doesn't hurt to
check the status of every dict lookup…

[jbenet]

I haven't tested it, but I see that this is a hole.

1. bugs or changes can creep in and violate assumptions
2. the CLI may be used against any process implementing the IPFS API (+/- bugs there too). there is zero guarantee it is the same code that is in go-ipfs.
3. safety first :) -- the data ought to be validated.

[wking]

On Tue, Jun 16, 2015 at 04:31:31PM -0700, Juan Batiz-Benet wrote:

1. the CLI may be used against any process implementing the IPFS
   API. there is zero guarantee it is the same code that is in
   go-ipfs.

Where does that split land? I'd expect both the Run and Marshalers
handling to happen on the daemon side, so they need to be in sync with
each other, but not necessarily with any client code.

I'll reroll to check the map-access (since I understand points (1) and
(3)), but if there's some important kernel of wisdom in point (2), I'd
like to clear up my confusion there ;).

[jbenet]

i just mean that there is a json API, and other implementations may match it, but have problems. so even though the Run and Marshalers are related in go-ipfs, it does not mean the cli will always be run against a go-ipfs daemon

[jbenet]

Comments above o/

[wking]

I just pushed fixes for the single directory listing 1 and the JSON
restructuring to avoid abusing LsLink 2. Still to go:

• Adding missing 'ok' checks 3.

[jbenet]

@wking this LGTM! thanks. once the ok check is done, this is RFM, i think

[wking]

On Thu, Jun 18, 2015 at 05:28:35PM -0700, Juan Batiz-Benet wrote:

once the ok check is done, this is RFM, i think

I added an ok check for output.Objects[hash] and shifted the
output.Arguments value extraction into the range call. These changes
seemed unambiguous enough so I squashed them back into faf83f6
(core/commands/unixfs/ls: Hash-map for Objects, 2015-06-13) instead of
adding them as a new commit.

[jbenet]

@wking LGTM. I'll run gpe.

[whyrusleeping]

@jbenet every commit appears to have passed with GPE. I'll merge

[jbenet]

yep, LGTM.