Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Runs to fast for a free infura account #2

Closed
DarkGhost7 opened this issue Nov 15, 2020 · 2 comments
Closed

Runs to fast for a free infura account #2

DarkGhost7 opened this issue Nov 15, 2020 · 2 comments
Labels
enhancement New feature or request

Comments

@DarkGhost7
Copy link
Contributor

Running it at its current speed will use up a free 100k requests infura account in a few hours.
@banteg
Copy link
Member

banteg commented Dec 5, 2020

This is known, optimizing it to work with Infura would require a bit more than adding a sleep after each iteration.

I will address this by batching calls using multicall and adding an optional sleep interval.

@banteg banteg added the enhancement New feature or request label Dec 5, 2020
rudygt pushed a commit to rudygt/yearn-exporter that referenced this issue Feb 14, 2021
@mariuspod
Merge pull request yearn#2 from DarkGhost7/Add_aave_3pool_v1_vault
327e037 
added dashboards
@mariuspod
Copy link
Contributor

mariuspod commented Mar 9, 2021 •
edited
Loading

I think we've addressed it by using multicall and applying a sleep interval via env variable SLEEP_SECONDS set to e.g. 900 (15 minutes)

banteg added a commit that referenced this issue Nov 25, 2021
@BobTheBuidler @banteg
feat: wallet and treasury stats (#159)
a3b593f 
* feat: add tx exporter

* feat: add skeleton

* fix: temporarily disable victoria

* feat: add total users

* feat: count wallets by # vaults used

* feat: add total users by vault

* fix: use Decimal instead of float

* feat: use pickle instead of csv for tx cache

* chore: refactor

* chore: checkpoint tx pickle thru block 10949751

* chore: add pandas to requirements.txt

* chore: add env/ to .gitignore

* feat: parallelism

* fix: accept new transfer event field labels

* fix: handle reverts in price magic

* fix: etherscan rate limit handler

* chore: refactor dataframe

* chore: increase dop

* chore: refactor

* chore: checkpoint tx pickle thru block 13398262

* feat: add transactions exporter to docker

* feat:user balances (#2)

* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances

* feat: merge tx analysis with existing exporter

* chore: wallets not users

* chore: cleanup

* chore: wallets not users

* fix: tvl logger

* chore: refactor

* feat: env variable to skip wallet stats for speed

* chore: cleanup

* fix: skip transfers

* fix: typo

* feat:user balances (#2)

* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances

* feat:user balances (#2)

* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances

* feat: treasury exporter

* chore: refactor and minor fixes

* feat: add buckets

* fix: revert curve oracle to use common base token

* fix: curve underlying coins revert for non-metapools

* fix: add details to revert message when can't decode logs

* fix: historical treasury exporter start date

* feat:user balances (#2)

* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances

* fix: revert curve oracle to use common base token

* fix: curve underlying coins revert for non-metapools

* feat: postgres container

* chore: add pyodbc to requirements

* feat: initial postgres design

* feat: silent kwarg for magic.get_price

* feat: postgres tx caching

* chore: better logging for treasury

* feat: make logs

* chore: add sqlalchemy to requirements

* chore: cleanup reverted changes

* chore: specify containers for make logs

* feat: fetch balances from postgres

* feat: yearn.describe_wallets

* feat: wallet exporter

* chore: sort imports

* chore: formatting

* feat: specify new agg stats

* feat: skip middleware if already setup

* chore: cleanup

* chore: black

* chore: remove unused function

* chore: cleanup

* fix: remove rai from stablecoins

* chore: cleanup

* fix: setup middleware

* chore: revert changes to create_filter

* chore: revert silent kwarg on magic.get_price

* chore: refactor

* feat: prices for one to one wrapped tokens

* feat: add ymechs msig to treasury wallets

* feat: two more yearn wallets

* chore: ignore .crypto domain NFT transfers

Co-authored-by: banteg <[email protected]>
BobTheBuidler added a commit that referenced this issue Jan 18, 2022
@BobTheBuidler
feat:user balances (#2)
0a01240 
* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances
banteg added a commit that referenced this issue Jan 31, 2022
@BobTheBuidler @mariuspod @banteg
Wallets and treasury update (#182)
21ee091 
* feat: add skeleton

* fix: temporarily disable victoria

* feat: add total users

* feat: count wallets by # vaults used

* feat: add total users by vault

* fix: use Decimal instead of float

* feat: use pickle instead of csv for tx cache

* chore: refactor

* chore: checkpoint tx pickle thru block 10949751

* feat: parallelism

* fix: accept new transfer event field labels

* fix: handle reverts in price magic

* fix: etherscan rate limit handler

* chore: refactor dataframe

* chore: increase dop

* chore: refactor

* chore: checkpoint tx pickle thru block 13398262

* feat:user balances (#2)

* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances

* feat: merge tx analysis with existing exporter

* chore: wallets not users

* chore: cleanup

* chore: wallets not users

* chore: refactor

* feat: env variable to skip wallet stats for speed

* chore: cleanup

* fix: skip transfers

* fix: typo

* feat: treasury exporter

* feat: add buckets

* fix: add details to revert message when can't decode logs

* feat: silent kwarg for magic.get_price

* feat: postgres tx caching

* chore: cleanup reverted changes

* feat: yearn.describe_wallets

* feat: wallet exporter

* feat: specify new agg stats

* feat: skip middleware if already setup

* chore: cleanup

* chore: black

* chore: cleanup

* chore: cleanup

* fix: setup middleware

* chore: revert changes to create_filter

* chore: revert silent kwarg on magic.get_price

* chore: refactor

* feat: sms exporter

* fix: compound debt zero address bug

* chore: fix logger name

* feat: replace pyodbc with ponyorm

* feat: wallet exporter for earn, ib, special

* chore: add pony to requirements

* fix: sms exporter

* fix: wallet exporter

* fix: historical treasury exporter

* fix: treasury exporter

* fix: sql syntax

* fix: registry wallet exporter

* fix: vault wallet exporter

* fix: yearn registry

* fix: victoria

* fix: transactions exporter insufficient chunk size

* feat: add sms address for ftm

* fix: typeerror in compound debt calc

* chore: cleaner treasury constants for multichain

* chore: homogenize registry method names for active_vaults_at

* chore: remove unnecessary caching

* chore: cleanup constants

* chore: refactor

* chore: remove artifact from testing

* chore: better utilize ponyorm queries

* chore: refactor

* fix: last_recorded_block postgres util

* fix: transactions exporter broken by previous refactoring

* chore: remove artifact from testing

* chore: remove unneeded infura env vars

* chore: remove dev artifacts

* fix: remove parallelism on transactions exporter

* chore: extract variable

* chore: add comment

* chore: refactor

* chore: remove commented out wip snippets

* fix: buckets

* fix: unhashable type error

* chore: clarify comment

* fix: stablecoin checksum addresses

* feat: add husd, dusd, musd, fusdt(ftm), mim (ftm) to stablecoin list

* feat: support yearn vaults on ftm price magic

* chore: remove old code

* fix: magic.get_price for yveCRV-DAO before there was liquidity

* fix: data_query for ftm historical exporter

* chore: remove empty exports

* chore: remove unused imports

* chore: cleanup logging

* feat: magic.get_price for curve v2

* chore: improve exception logging

* fix: price edge cases for wallet exporter

* fix: null data display bug

* Revert "feat: magic.get_price for curve v2"

This reverts commit 15e3d1d.

* feat: add anydai and anymim to mainnet stablecoins list

* fix: import error when postgres container not running

* feat: log errors while fetching price to sentry

* feat: more responsive treasury event watching

* fix: treasury watch events

* fix: manually merge ib handling code into the PR branch.

* feat: refactored outputs into separate files for better readability.

* feat: adding treasury file from bob.

* fix: move import to the right file

* feat: remove unused packages from docker image.

* chore: add FTMSCAN_TOKEN env var for ftm containers

* fix: remove IS_CONTAINER env var

* fix: transactions-exporter TransactionIntegrityError on fresh startup

* chore: replace soon-to-be-deprecated df.append with pd.concat

* fix: don't fail when transaction_exporter has not exported anything yet.

* fix: handle the entity creation more gracefully with the transaction so it doesn't raise IntegrityException.

* feat: refactor db entities a bit for more standardized PKs and FKs and less ORM surprises.

* fix: adding band supported tokens.

* fix: use PG* envs for postgres connection config.

* wip: trying to move pricePerShare logic into its own price class.

* fix: adding file to calculate price per share of a token

* Revert "fix: adding file to calculate price per share of a token"

This reverts commit d215e97.

* Revert "wip: trying to move pricePerShare logic into its own price class."

This reverts commit 3efcdba.

* fix: propagate error when unable to read share price

* fix: handle crvaave incident

Co-authored-by: mariuspod <[email protected]>
Co-authored-by: banteg <[email protected]>
BobTheBuidler added a commit that referenced this issue Feb 1, 2022
@BobTheBuidler
feat:user balances (#2)
6416766 
* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances
BobTheBuidler added a commit that referenced this issue Feb 4, 2022
@BobTheBuidler
feat:user balances (#2)
1293eee 
* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances
BobTheBuidler added a commit that referenced this issue Feb 5, 2022
@BobTheBuidler
feat:user balances (#2)
91f57e9 
* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances
BobTheBuidler added a commit that referenced this issue Feb 9, 2022
@BobTheBuidler
feat:user balances (#2)
aff46e3 
* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances
BobTheBuidler added a commit that referenced this issue Feb 9, 2022
@BobTheBuidler
feat:user balances (#2)
1b95d37 
* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances
BobTheBuidler added a commit that referenced this issue Feb 10, 2022
@BobTheBuidler
feat:user balances (#2)
1fd268f 
* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances
BobTheBuidler added a commit that referenced this issue Mar 1, 2022
@BobTheBuidler
feat:user balances (#2)
aa1a87b 
* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances
BobTheBuidler added a commit that referenced this issue Mar 10, 2022
@BobTheBuidler
[EXP-31-231] Transactions and wallets ftm (#231)
02a7b7f 

* fix: accept new transfer event field labels

* fix: handle reverts in price magic

* fix: etherscan rate limit handler

* chore: refactor dataframe

* chore: increase dop

* chore: refactor

* chore: checkpoint tx pickle thru block 13398262

* feat:user balances (#2)

* feat: add transactions exporter to docker

* chore: reduce unnecessary printing

* feat: user balances

* feat: merge tx analysis with existing exporter

* chore: wallets not users

* chore: cleanup

* chore: wallets not users

* chore: refactor

* feat: env variable to skip wallet stats for speed

* chore: cleanup

* fix: skip transfers

* fix: typo

* feat: treasury exporter

* feat: add buckets

* fix: add details to revert message when can't decode logs

* feat: postgres tx caching

* chore: cleanup reverted changes

* feat: yearn.describe_wallets

* feat: specify new agg stats

* chore: cleanup

* chore: black

* chore: cleanup

* chore: cleanup

* chore: revert changes to create_filter

* chore: revert silent kwarg on magic.get_price

* chore: refactor

* feat: sms exporter

* feat: replace pyodbc with ponyorm

* feat: wallet exporter for earn, ib, special

* fix: wallet exporter

* fix: vault wallet exporter

* fix: yearn registry

* fix: victoria

* chore: remove unnecessary caching

* chore: remove artifact from testing

* chore: better utilize ponyorm queries

* chore: remove dev artifacts

* chore: remove old code

* chore: remove unused imports

* fix: price edge cases for wallet exporter

* fix: null data display bug

* fix: fantom env vars

* chore: remove unused import

* feat: make logs-all

* fix: broken import

* feat: temporarily remove ftm transactions and wallet exporter

* chore: temporarily remove ftm transactions and wallet exporter from makefile

* chore: remove unused files

* chore: refactor

* feat: add ftm transactions and wallet exporter to make logs

* fix: add docker-compose postgres dependency for ftm-wallet-exporter

* fix: improper handling of rebase conflicts

* chore: remove unused import

* fix: add vm dependency to wallet exporters in docker-compose

* chore: remove old functions

* fix: rename magic.describe_err to magic._describe_err for intent clarity

* fix: transactions_exporter PriceError for yUSDC due to price magic refactor

* chore: remove unused vars

* chore: cleanup treasury exporter

* fix: list.remove error for transactions_exporter on Ygov()

* fix: move __add_deployment_topics from creat_filter to decode_ logs

* feat: add sentry tag 'script'

* fix: magic _describe_err

* fix: treasury _get_price AttirbuteError

* feat: move treasury and transactions exporter special price handling to separate file

* chore: improve magic readability

* feat: add anyusdc to fantom stablecoins

* fix: transactions exporter infinite loop on fantom

* feat: use chainlink eth feed for anyeth on fantom

* fix: yearn.active_vaults_at comparison error

* fix: treasury exporter where token.symbol fails

* fix: registry wallet describer for non-mainnet chains

* fix: AssertionError in fetch_multicall with require_success=False

* fix: decimals AssertionErrors in price magic

* chore: extract treasury nft list

* fix: UnboundLocalError

* fix: yvobtc v1 PriceError

* fix: egrigious price for yvpbtc v1

* fix: transactions_exporter infinite loop logging

* fix: start_block > end_block

* fix: transactions exporter infinite loop handler

* fix: transactions exporter end_block

* feat: who uses yearn dash

* feat: use chainlink for anybtc on ftm

* fix: specify block number for tvl exporter

* fix: get_token_bucket for non verified contracts

* feat: get_price for generic amm

* fix: get_price for non verified generic amm
jstashh added a commit that referenced this issue Apr 11, 2022
@jstashh @mariuspod
fix: more convex apy fixes (#284)
fafc2fd 
* fix: rk3pr pricing, correct order of strategies

* fix: fallback to id property if no pid

* chore: add clarifying comment

* chore: clarify comment, convex not curve

* refactor: remove duplication of reward token pricing logic

* fix: handle strategies with localKeepCRV variables

* fix: prefer hasattr over try except

* fix: add check for chain id

* refactor: extract addresses for chain into separate variable

* fix: remove duplicate variable name

* tests: started with some unit and integration tests for APY calculations. (#2)

Co-authored-by: mariuspod <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@banteg @mariuspod @DarkGhost7