[Stargate] Where is the Not Implemented error coming from?

[michaelfig]

Describe the bug

I'm trying to find out where the following log message is being caused:

Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]: This block used: 1.463seconds.
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]: Getting block at height: 238
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]: Error: failed [501] {   "code": 12,   "message": "Not Implemented",   "details": [   ] }
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Object.exports.makeErrorByStatus (packages/http.js:182:10)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Request._callback (packages/http.js:142:24)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Request.self.callback (/home/bigdipper/portal-stargate-agorictest-6/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:185:22)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Request.emit (events.js:314:20)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Request.EventEmitter.emit (domain.js:483:12)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Request.<anonymous> (/home/bigdipper/portal-stargate-agorictest-6/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:1161:10)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Request.emit (events.js:314:20)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Request.EventEmitter.emit (domain.js:483:12)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at IncomingMessage.<anonymous> (/home/bigdipper/portal-stargate-agorictest-6/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:1083:12)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at Object.onceWrapper (events.js:420:28)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at IncomingMessage.emit (events.js:326:22)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at IncomingMessage.EventEmitter.emit (domain.js:483:12)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at endReadableNT (_stream_readable.js:1241:12)
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     at processTicksAndRejections (internal/process/task_queues.js:84:21) {
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:   response: {
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     statusCode: 501,
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     content: '{\n  "code": 12,\n  "message": "Not Implemented",\n  "details": [\n  ]\n}',
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     headers: {
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:       'content-type': 'application/json',
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:       'x-server-time': '1615860973',
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:       date: 'Tue, 16 Mar 2021 02:16:13 GMT',
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:       'content-length': '68',
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:       connection: 'close'
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     },
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:     data: { code: 12, message: 'Not Implemented', details: [] }
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]:   }
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]: }
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]: updateStatus: 355818
Mar 16 02:16:13 explorer-testnet bigdipper.sh[2203]: Get height time: 0.163seconds.

How can I enhance the log messages to show the query that's being issued?

To Reproduce
Steps to reproduce the behavior:

1. Here is my settings.json:

{
    "public":{
        "chainName": "Agoric Devnet",
        "chainId": "agorictest-6",
        "gtm": "<my GTM>",
        "slashingWindow": 10000,
        "uptimeWindow": 250,
        "initialPageSize": 30,
        "bech32PrefixAccAddr": "agoric",
        "bech32PrefixAccPub": "agoricpub",
        "bech32PrefixValAddr": "agoricvaloper",
        "bech32PrefixValPub": "agoricvaloperpub",
        "bech32PrefixConsAddr": "agoricvalcons",
        "bech32PrefixConsPub": "agoricvalconspub",
        "bondDenom": "uagstake",
        "powerReduction": 1000000,
        "coins": [
            {
                "denom": "uagstake",
                "displayName": "AGSTAKE",
                "fraction": 1000000
            },
 	    {
 		"denom": "uag",
 		"displayName": "AG",
 		"fraction": 1000000
 	    }
        ],
        "ledger": {
          "coinType": 564,
          "appName": "Cosmos",
          "appVersion": "2.16.0",
          "gasPrice": 0.00
        },
        "modules": {
          "bank": true,
          "supply": true,
          "minting": false,
          "gov": true,
          "distribution": false
        },
        "coingeckoId": null
    },
    "remote":{
        "rpc":"http://node0.testnet.agoric.com:26657",
        "api":"http://node0.testnet.agoric.com:1317"
    },
    "debug": {
        "startTimer": true
    },
    "params":{
        "startHeight": 0,
        "defaultBlockTime": 5000,
        "validatorUpdateWindow": 300,
        "blockInterval": 15000,
        "transactionsInterval": 18000,
        "keybaseFetchingInterval": 18000000,
        "consensusInterval": 1000,
        "statusInterval":7500,
        "signingInfoInterval": 1800000,
        "proposalInterval": 5000,
        "missedBlocksInterval": 60000,
        "delegationInterval": 900000
    }
}

2. Run Big Dipper commit: d3735e8

Expected behavior
I expected to see the URL that was causing the error so I could diagnose further.

Screenshots
n/a

Desktop (please complete the following information):
n/a

Smartphone (please complete the following information):
n/a

Additional context

I'm using [email protected]. will upgrade soon to v0.42.1

[kwunyeung]

@michaelfig thank you for the report. Let me take a look.

[kstoykov]

I can confirm that I have the same issue

[nebkas]

I had similar notifications. For me it was that full indexing of the rpc/api node wasn't enabled. After enabling full indexing on the node the notifications were gone. You could just setup a local full node and try to run BigDipper against that full node.

I'm running on a different network, but think it could be the same. Here the sample from the node config for full indexing:

#######################################################
###   Transaction Indexer Configuration Options     ###
#######################################################
[tx_index]

# What indexer to use for transactions
#
# The application will set which txs to index. In some cases a node operator will be able
# to decide which txs to index based on configuration set in the application.
#
# Options:
#   1) "null"
#   2) "kv" (default) - the simplest possible indexer, backed by key-value storage (defaults to levelDB; see DBBackend).
# 		- When "kv" is chosen "tx.height" and "tx.hash" will always be indexed.
indexer = "kv"