Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Towards Issue #875, avoid duplicating files added to ipfs #2600

Closed
wants to merge 32 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
6a187c0
Refactor Makefile.
kevina Apr 19, 2016
27ec622
Add Offset() method to files.File and create new AdvReader interface.
kevina Apr 20, 2016
c410ea8
Add AbsPath() to files.File interface.
kevina Apr 21, 2016
92c13ba
Add extra parameter to indicate how the content should be added.
kevina Apr 22, 2016
abfb0a6
In the measure package don't return ErrInvalidType in batch Put.
kevina Apr 21, 2016
e92e9da
In merkledag.Node and blocks.Block maintain a DataPtr
kevina Apr 21, 2016
d133b68
New "multi" datastore.
kevina Apr 22, 2016
1c91a03
Basic implementation of "add --no-copy".
kevina Apr 22, 2016
2b8704e
Add basic tests for "add --no-copy".
kevina Apr 22, 2016
f26c2df
When reconstructing block always verify the result.
kevina Apr 23, 2016
8ff6185
Bug fix to commit e92e9dac:
kevina Apr 27, 2016
955e763
Add Basic Query and "Direct" commands to filestore. Needs Testing.
kevina Apr 24, 2016
14601a2
Save ref to Filestore in repo for future direct access.
kevina Apr 24, 2016
31e2e21
Add Self() method to be able to get to FSRepo.
kevina Apr 25, 2016
4ef5531
Add "filestore" commands to list contents and verify filestore.
kevina Apr 25, 2016
a999a72
Make blocks.Block an interface.
kevina Apr 28, 2016
7addf3e
Remove addOpts paramater from blockstore methods.
kevina Apr 28, 2016
1685cbd
Disable failing test.
kevina Apr 27, 2016
12ff7ac
Refactor.
kevina Apr 27, 2016
01ad077
Add temp. utility command to find dangling pins.
kevina Apr 27, 2016
72966dc
Add "filestore rm" command.
kevina Apr 27, 2016
63127e5
"filestore ls": add "--quiet" option
kevina Apr 27, 2016
767d4c8
"filestore verify": change "invalid" status to "changed".
kevina Apr 27, 2016
437905e
Check if the WholeFile flag should be set for leaf nodes in filestore…
kevina Apr 27, 2016
a3a2c2a
Add "filestore rm-invalid" command.
kevina Apr 27, 2016
10d724a
"filestore ls": add help text and rework output
kevina Apr 28, 2016
dd6fbc0
"filestore verify": only verify leaf nodes, add help text
kevina Apr 29, 2016
dd46d8d
Add sharness tests for new "filestore" commands.
kevina Apr 29, 2016
13a14a6
Simpilfy files.File and chunk.Splitter interface.
kevina Apr 29, 2016
1e21dbe
Eliminate separate addOpts parameter by pushing the info into the rea…
kevina Apr 29, 2016
993a6d6
Remove ExtraInfo() from File interface and add SetExtraInfo() to AdvR…
kevina Apr 29, 2016
56a6b43
Report an error when trying to use "--no-copy" when daemon is online.
kevina Apr 30, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

204 changes: 204 additions & 0 deletions Godeps/_workspace/src/github.com/ipfs/go-datastore/multi/multi.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 5 additions & 7 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ endif

COMMIT := $(shell git rev-parse --short HEAD)
ldflags = "-X "github.com/ipfs/go-ipfs/repo/config".CurrentCommit=$(COMMIT)"
MAKEFLAGS += --no-print-directory


export IPFS_API ?= v04x.ipfs.io

Expand Down Expand Up @@ -45,19 +43,19 @@ vendor: godep
godep save -r ./...

install: deps
cd cmd/ipfs && go install -ldflags=$(ldflags)
make -C cmd/ipfs install

build: deps
cd cmd/ipfs && go build -i -ldflags=$(ldflags)
make -C cmd/ipfs build

nofuse: deps
cd cmd/ipfs && go install -tags nofuse -ldflags=$(ldflags)
make -C cmd/ipfs nofuse

clean:
cd cmd/ipfs && go clean -ldflags=$(ldflags)
make -C cmd/ipfs clean

uninstall:
cd cmd/ipfs && go clean -i -ldflags=$(ldflags)
make -C cmd/ipfs uninstall

PHONY += all help godep toolkit_upgrade gx_upgrade gxgo_upgrade gx_check
PHONY += go_check deps vendor install build nofuse clean uninstall
Expand Down
52 changes: 41 additions & 11 deletions blocks/blocks.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,39 +12,69 @@ import (
)

// Block is a singular block of data in ipfs
type Block struct {
Multihash mh.Multihash
Data []byte
type Block interface {
Multihash() mh.Multihash
Data() []byte
Key() key.Key
String() string
Loggable() map[string]interface{}
}

type BasicBlock struct {
multihash mh.Multihash
data []byte
}

type FilestoreBlock struct {
BasicBlock
*DataPtr
AddOpts interface{}
}

// This DataPtr had different AltData than the node DataPtr
type DataPtr struct {
AltData []byte
FilePath string
Offset uint64
Size uint64
}

// NewBlock creates a Block object from opaque data. It will hash the data.
func NewBlock(data []byte) *Block {
return &Block{Data: data, Multihash: u.Hash(data)}
func NewBlock(data []byte) *BasicBlock {
return &BasicBlock{data: data, multihash: u.Hash(data)}
}

// NewBlockWithHash creates a new block when the hash of the data
// is already known, this is used to save time in situations where
// we are able to be confident that the data is correct
func NewBlockWithHash(data []byte, h mh.Multihash) (*Block, error) {
func NewBlockWithHash(data []byte, h mh.Multihash) (*BasicBlock, error) {
if u.Debug {
chk := u.Hash(data)
if string(chk) != string(h) {
return nil, errors.New("Data did not match given hash!")
}
}
return &Block{Data: data, Multihash: h}, nil
return &BasicBlock{data: data, multihash: h}, nil
}

func (b *BasicBlock) Multihash() mh.Multihash {
return b.multihash
}

func (b *BasicBlock) Data() []byte {
return b.data
}

// Key returns the block's Multihash as a Key value.
func (b *Block) Key() key.Key {
return key.Key(b.Multihash)
func (b *BasicBlock) Key() key.Key {
return key.Key(b.multihash)
}

func (b *Block) String() string {
func (b *BasicBlock) String() string {
return fmt.Sprintf("[Block %s]", b.Key())
}

func (b *Block) Loggable() map[string]interface{} {
func (b *BasicBlock) Loggable() map[string]interface{} {
return map[string]interface{}{
"block": b.Key().String(),
}
Expand Down
Loading