feat: filter pins by status (#848)

* feat(pinning-api): add skeleton and validation for pinning apis * feat: add pinning endpoints * feat: add tests * feat: add validation for endpoints and refactor error messages * fix: validation bug * feat: add tests for get and delete endpoints * fix: test assertions * feat: add replace pin api endpoint * chore: rollback type changes for now Co-authored-by: Paolo <[email protected]> * feat(pinning-apis): work on db to support new pinning apis * chore: rename PinsUpsertInput to PinUpsertInput * chore: fix PinUpsertInput type * chore: use PinUpsertInput where required * chore(refactor types): refactor pinItem * chore(refactor types): Use location with other types * chore(refactor types): fix some type errors in db client * chore(refactor types): rename PinItemOutput to PinItemNormalized * chore(refactor types): Pin item should have an _id field * Rename PinItemOutput back * feat(pinning-api-db): create pinning table and update reset.sql * feat(pinning-api-db): create initial types * feat(pinning-api-db): create db client signatures * feat(pinning-api-db): updat types * feat(pinning-api-db): write first intial add pin spec * feat(pinning-api-db): first implementation of create and get pinRequest * feat(pinning-api-db): get pin data for request * feat(pinning-api-db): housekeeping * feat(pinning-api-db): get pin request tests * feat(pinning-api-db): housekeeping * feat: add types to db data types * feat(pinning-api-db): create content function * feat(pinning-api-db): remove duplicated type * feat(pinning-api-db): housekeeping * feat(pinning-api-db): update types * feat(pinning-api-db): update documentation * Add endpoint and logic to our api service (#702) * feat(pinning-apis): get request endpoint work * chore: remove logging * chore: improve validation get /pin/requestId * chore: better integer validation * fix: requestId conditional * fix: remove extra bracket Co-authored-by: Alexandra Stoica <[email protected]> * feat(pinning-apis): POST /pin endpoint * wip: create pin request * feat: add async task * chore: merge feature branch * feat: code improvements * chore: add todo * feat: add comments and some small fixes * fix: pass token id * fix: normalized cid vs source cid * fix: minor fix and tests updates * fix: update return of get user mock Co-authored-by: Paolo <[email protected]> * chore: make standard happier * fix: fix token referemce * fix: merge conflict issue * feat : delete and replace pin requests endpoints * wip: delete pin request * feat: add db definitions * feat: wip replace pin endpoint * chore: update mocks * fix: do not list deleted requests * feat: add delete db tests * fix: tests * fix: improve validation * feat: tests * chore: update comments * fix: replace pin bugs and tests * fix: coerce to number using parseInt * chore: disable tests for now Co-authored-by: Paolo <[email protected]> * feat: GET /pins endpoint * feat: wip list pin requests * feat: list pinning requests * chore: renaming test descriptions * chore: filter for list pins * chore: fixing tests * chore: list pins tests * feat: db and api tests for list pins * chore: reorder list test * fix: correct count in list response * fix: avoid double query on list * feat: PR feedback * feat: add todo for future improv * fix: typos, use new URL in test, improve Date test * feat: don't skip test * fix: update tests and more fixes * fix: take back the todo Co-authored-by: Gary Homewood <[email protected]> * feat: filter pins by status * Refactor Pin APIs and PR feedback (#810) chore: pinning apis refactoring and feedback * feat: update sql and db client logic * feat: update tests and minor fixes * feat: more updates/fixes to db package * feat: refactor pin creation flow and refactor car upload as well * feat: pinning, fix tests and add more * fix: rename table, functions and types * chore: PR feedback * fix: rename types * chore: rename requestedCid to sourceCid * fix: delete should not return a body and status should be 202 * chore: add some documentation to waitOkPins * chore: delete stale mock * fix: do not passa meta to cluster * chore: remove comment * chore: refactor createPin * chore: linting * fix: updates and fixes after merge * Get request id by user token (#863) * fix: get pin req ids by user token * fix: update tests * fix: rest api types * fix: update function params * fix: type name * feat: add test * fix: tests * fix: merge conflict * fix: merge * fix: rename authToken id * fix: requestedCid to sourceCid * chore: clean up fixtures * chore: refactor filter by status in the db package * chore: refactor pinning api to match new db api * chore: use chromium for tests * chore: remove console log * Add metadata to API (#881) * feat: add metadata to psa pin req * feat: add metadata to api * feat: add metadata tests * chore: feedback * feat: added pinning service request issue mailto link/github button (#879) * feat: added pinning service request issue mailto link/github button * chore: new line * Fix test * fix: update comment to reflect changes * fix: lint error Co-authored-by: Alexandra Stoica <[email protected]> Co-authored-by: Paolo <[email protected]> Co-authored-by: Paolo Chillari <[email protected]> Co-authored-by: Alexandra Stoica <[email protected]> Co-authored-by: Leslie Owusu-Appiah <[email protected]>
web3-storage · Jan 26, 2022 · df1582b · df1582b
1 parent 0056679
commit df1582b
Show file tree

Hide file tree

Showing 7 changed files with 297 additions and 108 deletions.
diff --git a/packages/api/src/pins.js b/packages/api/src/pins.js
@@ -62,6 +62,28 @@ export const getEffectivePinStatus = (pins) => {
   return 'failed'
 }
 
+const psaStatusesToDBStatusesMap = {
+  pinned: ['Pinned'],
+  queued: ['PinQueued'],
+  pinning: ['Pinning'],
+  failed: [
+    'ClusterError',
+    'PinError',
+    'Unpinned'
+  ]
+}
+
+/**
+ * Maps a pinning api status array to db status accepted by the DB
+ * @param {string[]} statuses
+ * @return {import('@web3-storage/db/postgres/pg-rest-api-types').definitions['pin']['status'][]}
+ */
+const psaStatusesToDBStatuses = (statuses) => {
+  return statuses.reduce((mappedStatuses, psaStatus) => {
+    return mappedStatuses.concat(psaStatusesToDBStatusesMap[psaStatus])
+  }, [])
+}
+
 // Error messages
 // TODO: Refactor errors
 export const ERROR_CODE = 400
@@ -342,16 +364,15 @@ function parseSearchParams (params) {
 
   if (status) {
     const statuses = status.split(',')
-    const isValidStatus = status.every(v => STATUS_OPTIONS.includes(v))
+    const isValidStatus = statuses.every(v => STATUS_OPTIONS.includes(v))
 
     if (!isValidStatus) {
       return {
         error: { reason: ERROR_STATUS, details: INVALID_STATUS },
         data: undefined
       }
     }
-    // TODO(https://github.com/web3-storage/web3.storage/issues/797): statuses need to be mapped to db statuses
-    opts.status = statuses
+    opts.statuses = psaStatusesToDBStatuses(statuses)
   }
 
   if (before) {

diff --git a/packages/api/test/fixtures/init-data.sql b/packages/api/test/fixtures/init-data.sql
@@ -79,7 +79,15 @@ VALUES  ('bafybeid46f7zggioxjm5p2ze2l6s6wbqvoo4gzbdzfjtdosthmfyxdign4'),
         ('bafybeifsrhq2qtkcgjt4gzi7rkafrv2gaai24ptt6rohe2ebqzydkz47sm'),
         ('bafybeiaqu6ijhfhwzjipwesbqf4myz6uczyigahib5joqbo5jw2xmjczfa'),
         ('bafybeidqts3rbwkprggjojbvcxy4jzpgzgcvs4a73y3gx2jjxphjeerbcy');
-
+
+INSERT INTO pin (status, content_cid, pin_location_id, inserted_at, updated_at)
+VALUES
+       ('Pinned', 'bafybeid46f7zggioxjm5p2ze2l6s6wbqvoo4gzbdzfjtdosthmfyxdign4', 1, '2021-07-14T19:27:14.934572+00:00', '2021-07-14T19:27:14.934572+00:00'),
+       ('Pinning', 'bafybeig7yvw6a4uhio4pmg5gahyd2xumowkfljdukad7pmdsv5uk5zcseu', 1, '2021-07-14T19:27:14.934572+00:00', '2021-07-14T19:27:14.934572+00:00'),
+       ('PinError', 'bafybeia45bscvzxngto555xsel4gwoclb5fxd7zpxige7rl3maoleznswu', 1, '2021-07-14T19:27:14.934572+00:00', '2021-07-14T19:27:14.934572+00:00'),
+       ('Pinning', 'bafybeidw7pc6nvm7u4rfhpctac4qgtpmwxapw4duugvsl3ppivvzibdlgy', 1, '2021-07-14T19:27:14.934572+00:00', '2021-07-14T19:27:14.934572+00:00'),
+       ('Pinning', 'bafybeidrzt6t4k25qjeasydgi3fyh6ejos5x4d6tk2pdzxkb66bkomezy4', 1, '2021-07-14T19:27:14.934572+00:00', '2021-07-14T19:27:14.934572+00:00'),
+       ('Pinning', 'bafybeifsrhq2qtkcgjt4gzi7rkafrv2gaai24ptt6rohe2ebqzydkz47sm', 1, '2021-07-14T19:27:14.934572+00:00', '2021-07-14T19:27:14.934572+00:00');
 
 INSERT INTO psa_pin_request (id, auth_key_id, content_cid, source_cid, name, origins, meta, inserted_at, updated_at)
 VALUES ('ab62cf3c-c98d-494b-a756-b3a3fb6ddcab', 3, 'bafybeid46f7zggioxjm5p2ze2l6s6wbqvoo4gzbdzfjtdosthmfyxdign4', 'bafybeid46f7zggioxjm5p2ze2l6s6wbqvoo4gzbdzfjtdosthmfyxdign4', 'ReportDoc.pdf', '["/ip6/2606:4700:60::6/tcp/4009/p2p/QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N", "/ip4/172.65.0.13/tcp/4009/p2p/QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx4N"]', null, '2021-07-14T19:27:14.934572Z', '2021-07-14T19:27:14.934572Z'),

diff --git a/packages/api/test/pin.spec.js b/packages/api/test/pin.spec.js
@@ -139,7 +139,7 @@ describe('Pinning APIs endpoints', () => {
       assert.strictEqual(error.details, INVALID_CID)
     })
 
-    it('returns the pins for this user with default filter values', async () => {
+    it('returns only successful pins when no filter values are specified', async () => {
       const res = await fetch(
         baseUrl, {
           method: 'GET',
@@ -152,13 +152,13 @@ describe('Pinning APIs endpoints', () => {
       assert(res, 'Server responded')
       assert(res.ok, 'Server response is ok')
       const data = await res.json()
-      assert.strictEqual(data.count, 6)
+      assert.strictEqual(data.count, 1)
     })
 
     it('filters pins on CID, for this user', async () => {
       const cids = [
-        'bafybeig7yvw6a4uhio4pmg5gahyd2xumowkfljdukad7pmdsv5uk5zcseu',
-        'bafybeia45bscvzxngto555xsel4gwoclb5fxd7zpxige7rl3maoleznswu',
+        'bafybeid46f7zggioxjm5p2ze2l6s6wbqvoo4gzbdzfjtdosthmfyxdign4', // Pinned
+        'bafybeia45bscvzxngto555xsel4gwoclb5fxd7zpxige7rl3maoleznswu', // PinError
         'bafybeiaiipiibr7aletbbrzmpklw4l5go6sodl22xs6qtcqo3lqogfogy4' // Not exists
       ]
 
@@ -219,6 +219,48 @@ describe('Pinning APIs endpoints', () => {
       assert.strictEqual(data.count, 3)
     })
 
+    it('filters pins by status', async () => {
+      const opts = new URLSearchParams({
+        status: 'failed'
+      })
+      const url = new URL(`${baseUrl}?${opts}`).toString()
+      const res = await fetch(
+        url, {
+          method: 'GET',
+          headers: {
+            Authorization: `Bearer ${token}`,
+            'Content-Type': 'application/json'
+          }
+        })
+
+      assert(res, 'Server responded')
+      assert(res.ok, 'Server response is ok')
+      const data = await res.json()
+      assert.strictEqual(data.count, 1)
+      assert.strictEqual(data.results.length, 1)
+      assert.strictEqual(data.results[0].pin.name, 'FailedPinning.doc')
+    })
+
+    it('filters pins by multiple status', async () => {
+      const opts = new URLSearchParams({
+        status: 'queued,pinning'
+      })
+      const url = new URL(`${baseUrl}?${opts}`).toString()
+      const res = await fetch(
+        url, {
+          method: 'GET',
+          headers: {
+            Authorization: `Bearer ${token}`,
+            'Content-Type': 'application/json'
+          }
+        })
+
+      assert(res, 'Server responded')
+      assert(res.ok, 'Server response is ok')
+      const data = await res.json()
+      assert.strictEqual(data.count, 4)
+    })
+
     it('filters pins created before a date', async () => {
       const opts = new URLSearchParams({
         before: '2021-07-01T00:00:00.000000Z'
@@ -456,14 +498,12 @@ describe('Pinning APIs endpoints', () => {
   })
 
   describe('GET /pins/:requestId', () => {
-    let pinRequest
-
     before(async () => {
       // Create token
       token = await getTestJWT('test-upload', 'test-upload')
 
       const cid = 'bafybeihy6bymmfcdjdrkhaha2srphnhrewimtkdxdmcama2dpgvpyx4efu'
-      pinRequest = await (await fetch(new URL('pins', endpoint).toString(), {
+      await (await fetch(new URL('pins', endpoint).toString(), {
         method: 'POST',
         headers: {
           Authorization: `Bearer ${token}`
@@ -501,7 +541,7 @@ describe('Pinning APIs endpoints', () => {
 
     it('returns not found if the request does not belong to the user token', async () => {
       const wrongToken = await getTestJWT()
-      const res = await fetch(new URL(`pins/${pinRequest.requestId}`, endpoint).toString(), {
+      const res = await fetch(new URL('pins/ab62cf3c-c98d-494b-a756-b3a3fb6ddcab', endpoint).toString(), {
         method: 'GET',
         headers: { Authorization: `Bearer ${wrongToken}` }
       })
@@ -511,7 +551,8 @@ describe('Pinning APIs endpoints', () => {
     })
 
     it('returns the pin request', async () => {
-      const res = await fetch(new URL(`pins/${pinRequest.requestId}`, endpoint).toString(), {
+      const requestId = 'ab62cf3c-c98d-494b-a756-b3a3fb6ddcab'
+      const res = await fetch(new URL(`pins/${requestId}`, endpoint).toString(), {
         method: 'GET',
         headers: {
           Authorization: `Bearer ${token}`,
@@ -523,8 +564,9 @@ describe('Pinning APIs endpoints', () => {
       assert(res, 'Server responded')
       assert(res.ok, 'Server response is ok')
       assertCorrectPinResponse(data)
-      assert.deepEqual(data.requestId, pinRequest.requestId)
-      assert.deepEqual(data.status, 'pinning')
+      assert.strictEqual(data.requestId, requestId)
+      assert.strictEqual(data.status, 'pinned')
+      assert.strictEqual(data.pin.cid, 'bafybeid46f7zggioxjm5p2ze2l6s6wbqvoo4gzbdzfjtdosthmfyxdign4')
     })
 
     it('returns the pin request with specified name', async () => {

diff --git a/packages/api/test/status.spec.js b/packages/api/test/status.spec.js
@@ -8,31 +8,31 @@ import statusWithNoDeal from './fixtures/pgrest/status-with-no-deal.json'
 describe('GET /status/:cid', () => {
   it('get pin and deal status', async () => {
     const cid = 'bafybeifnfkzjeohjf2dch2iqqpef3bfjylwxlcjws2msvdfyze5bvdprfm'
-    const res = await fetch(new URL(`status/${cid}`, endpoint))
+    const res = await fetch(new URL(`status/${cid}`, endpoint).toString())
     assert(res.ok, `${JSON.stringify(res)}`)
     const json = await res.json()
     assert.deepStrictEqual(json, statusWithActiveDeal)
   })
 
   it('get shows initial queued deal', async () => {
     const cid = 'bafybeica6klnrhlrbx6z24icefykpbwyypouglnypvnwb5esdm6yzcie3q'
-    const res = await fetch(new URL(`status/${cid}`, endpoint))
+    const res = await fetch(new URL(`status/${cid}`, endpoint).toString())
     assert(res.ok)
     const json = await res.json()
     assert.deepStrictEqual(json, statusWithQueuedDeal)
   })
 
   it('get shows no deals before aggregate is ready', async () => {
     const cid = 'bafybeiaiipiibr7aletbbrzmpklw4l5go6sodl22xs6qtcqo3lqogfogy4'
-    const res = await fetch(new URL(`status/${cid}`, endpoint))
+    const res = await fetch(new URL(`status/${cid}`, endpoint).toString())
     assert(res.ok)
     const json = await res.json()
     assert.deepStrictEqual(json, statusWithNoDeal)
   })
 
   it('get 404 for unknown cid', async () => {
     const cid = 'bafybeihgrtet4vowd4t4iqaspzclxajrwwsesur7zllkahrbhcymfh7kyi'
-    const res = await fetch(new URL(`status/${cid}`, endpoint))
+    const res = await fetch(new URL(`status/${cid}`, endpoint).toString())
     assert(!res.ok)
     assert.strictEqual(res.status, 404)
   })

diff --git a/packages/db/db-client-types.ts b/packages/db/db-client-types.ts
@@ -296,7 +296,7 @@ export type ListPsaPinRequestOptions = {
   /**
    * status  to match
    */
-  status?: Array<definitions['pin']['status']>
+  statuses?: Array<definitions['pin']['status']>
   /**
    * Uploads created before a given timestamp.
    */

diff --git a/packages/db/index.js b/packages/db/index.js
@@ -1,6 +1,8 @@
 import { PostgrestClient } from '@supabase/postgrest-js'
 
-import { normalizeUpload, normalizeContent, normalizePins, normalizeDeals, normalizePsaPinRequest } from './utils.js'
+import {
+  normalizeUpload, normalizeContent, normalizePins, normalizeDeals, normalizePsaPinRequest
+} from './utils.js'
 import { DBError } from './errors.js'
 import {
   getUserMetrics,
@@ -32,7 +34,31 @@ const pinRequestSelect = `
   deleted:deleted_at,
   created:inserted_at,
   updated:updated_at,
-  content(cid, dagSize:dag_size, pins:pin(status, updated:updated_at, location:pin_location(_id:id::text, peerId:peer_id, peerName:peer_name, region)))  `
+  content(cid, dagSize:dag_size, pins:pin(status, updated:updated_at, location:pin_location(_id:id::text, peerId:peer_id, peerName:peer_name, region)))`
+
+const listPinsQuery = `
+  _id:id::text,
+  sourceCid:source_cid,
+  contentCid:content_cid,
+  authKey:auth_key_id,
+  name,
+  deleted:deleted_at,
+  created:inserted_at,
+  updated:updated_at,
+  content!inner(
+    cid,
+    dagSize:dag_size,
+    pins:pin!inner(
+      status,
+      updated:updated_at,
+      location:pin_location(
+        _id:id,
+        peerId:peer_id,
+        peerName:peer_name,
+        region
+      )
+    )
+  )`
 
 /**
  * @typedef {import('./postgres/pg-rest-api-types').definitions} definitions
@@ -840,41 +866,46 @@ export class DBClient {
 
     let query = this._client
       .from(psaPinRequestTableName)
-      .select(pinRequestSelect)
+      .select(listPinsQuery)
       .eq('auth_key_id', authKey)
       .is('deleted_at', null)
       .order('inserted_at', { ascending: false })
 
-    if (opts.status) {
-      query = query.in('content.pins.status', opts.status)
-    }
-
-    if (opts.cid) {
-      query = query.in('source_cid', opts.cid)
-    }
-
-    if (opts.name && match === 'exact') {
-      query = query.like('name', `${opts.name}`)
-    }
-
-    if (opts.name && match === 'iexact') {
-      query = query.ilike('name', `${opts.name}`)
-    }
+    if (!Object.keys(opts).length) {
+      query = query.eq('content.pins.status', 'Pinned')
+    } else {
+      if (opts.statuses) {
+        query = query.in('content.pins.status', opts.statuses)
+      }
 
-    if (opts.name && match === 'partial') {
-      query = query.like('name', `%${opts.name}%`)
-    }
+      if (opts.cid) {
+        query = query.in('source_cid', opts.cid)
+      }
 
-    if (opts.name && match === 'ipartial') {
-      query = query.ilike('name', `%${opts.name}%`)
-    }
+      if (opts.name) {
+        switch (match) {
+          case 'exact':
+            query = query.like('name', `${opts.name}`)
+            break
+          case 'iexact':
+            query = query.ilike('name', `${opts.name}`)
+            break
+          case 'partial':
+            query = query.like('name', `%${opts.name}%`)
+            break
+          case 'ipartial':
+            query = query.ilike('name', `%${opts.name}%`)
+            break
+        }
+      }
 
-    if (opts.before) {
-      query = query.lte('inserted_at', opts.before)
-    }
+      if (opts.before) {
+        query = query.lte('inserted_at', opts.before)
+      }
 
-    if (opts.after) {
-      query = query.gte('inserted_at', opts.after)
+      if (opts.after) {
+        query = query.gte('inserted_at', opts.after)
+      }
     }
 
     // TODO(https://github.com/web3-storage/web3.storage/issues/798): filter by meta is missing