Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support upserts #292

Closed
iand opened this issue Dec 2, 2020 · 2 comments · Fixed by #324
Closed

Support upserts #292

iand opened this issue Dec 2, 2020 · 2 comments · Fixed by #324
Labels
kind/enhancement Improvement to an existing feature P2 P2: Should be resolved

Comments

@iand
Copy link
Contributor

iand commented Dec 2, 2020

Description

Currently all Visor models are inserted using ON CONFLICT DO NOTHING which discards the data if it conflicts with existing data. This is fine for appending new data but unsatisfactory when backfilling after changes to extraction logic or bug fixes. Visor should support upserting of data if a conflict is encountered. We may want to make this an optional flag for use when backfilling since in normal operation all writes are appends.

Acceptance criteria

  • Visor can be run in a mode where rows are updated if their primary key already exists in the database

Where to begin

  • Investigate go-pg support for upserting. If no built in support then change the OnConflict clause call in each model's persist method to update the non primary key columns.
@iand iand added kind/enhancement Improvement to an existing feature P2 P2: Should be resolved labels Dec 2, 2020
@frrist
Copy link
Member

frrist commented Dec 3, 2020

From the looks of go-pg documentation: https://pkg.go.dev/github.com/go-pg/pg/v10#example-DB.Model-InsertOnConflictDoUpdate it doesn't have "easy" support for upsert.

For comparison, GORM allows upserts to be performed via: https://gorm.io/docs/create.html#Upsert-On-Conflict

@frrist
Copy link
Member

frrist commented Dec 3, 2020

As an example, will probably go with something like?

func (a *Actor) PersistWithTx(ctx context.Context, tx *pg.Tx, policy model.ConflictPolicy) error {
	ctx, span := global.Tracer("").Start(ctx, "Actor.PersistWithTx")
	defer span.End()

	q := tx.ModelContext(ctx, a)

	if policy == model.DoNothing {
		if _, err := q.OnConflict("do nothing").
			Insert(); err != nil {
			return err
		}
	} else if policy == model.Upset {
		if _, err := q.OnConflict("(state_root)").
			Set("code = ?code, head = ?head, balance = ?balance, nonce = ?nonce").
			Insert(); err != nil {
			return err
		}
	} else {
		return xerrors.Errorf("unknown conflict policy: %d", policy)
	}

	return nil
}

ConflictPolicy is just a constant we set when instantiating the database.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement Improvement to an existing feature P2 P2: Should be resolved
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants