[Application] Provider.Quest #208
Replies: 15 comments 3 replies
-
|
I submitted a microgrant application here to add some geo data to the project: filecoin-project/devgrants#266 I think it might be the first microgrant application ever! I'll do this eventually, but if I can get funded, I can do it right away. :-) |
Beta Was this translation helpful? Give feedback.
-
|
I'm now publishing data for "named clients" ... I can map client addresses to names and the system will calculate and publish data for those. Here's a new notebook to view the data (showing only the "Estuary" deals). You can toggle "All deals" on and off to see how the named client data compares. https://observablehq.com/@jimpick/miner-report-deals-named-clients-hourly?name=estuary
|
Beta Was this translation helpful? Give feedback.
-
|
Great news! I applied for a Filecoin "Next Step" devgrant and it was approved! "Accurate geolocation data for visualizations" filecoin-project/devgrants#266 I spent the last few days working on a good system for collecting up-to-date multiaddrs from the on-chain "miner info" data as well as continuously scanning the DHT. As a bonus, I'm doing DNS resolutions on the multiaddrs ... currently only for the on-chain data (nobody seems to be publishing dns-based multiaddrs to the DHT presently). The next step is to publish the latest DHT data to a Textile bucket - I should have that done today. Then I should have a complete set of IP addresses that is continually updated to do IP geo-location with. There are a few other things I'm going to do as well as part of the grant. |
Beta Was this translation helpful? Give feedback.
-
|
I'm now doing DNS lookups for DHT-based multiaddrs now too. They are being used, but my scripts weren't collecting many of them. |
Beta Was this translation helpful? Give feedback.
-
|
Nice! |
Beta Was this translation helpful? Give feedback.
-
|
Today I added some aggregations to count the number of times I've had successful DHT queries over a single day and multiday window ... this is just an optimization so I don't spend so much time waiting for entries that probably don't exist. Now the scanning script uses that list and it can do a full scan of the recently seen miners in the DHT in about 3 minutes. I also randomly scan miners that haven't been seen recently to discover any that have come back online. You can see it in action here: |
Beta Was this translation helpful? Give feedback.
-
|
Just added to Miner.Report … a new notebook where I take the latest on-chain libp2p addresses + the results from the latest DHT scan, and I extract all the IP addresses (IPv4 and IPv6) … including DNS resolutions for /dns4/ and /dns6/ multi-addresses => https://observablehq.com/@jimpick/miner-report-multiaddr-ip-tool?collection=@jimpick/miner-report |
Beta Was this translation helpful? Give feedback.
-
|
Here’s a quick https://miner.report update … I’ve done a bunch of behind-the-scenes work to optimize the various scanning processes (miner power, miner info, DHT, asks, etc.) to focus on the most-likely-to-be-available data first, followed by randomized “wider focus” scans to capture data from miners outside that initial focus. I decided to ignore miners that have zero power averages … so new miners will need to seal at least a sector before the system will pick them up. Yesterday, I built a scanner notebook that does GeoIP lookups for all of the IP addresses using the MaxMind GeoLite2 database. Today I will put that into Apache Spark and see if I can republish the data to a Textile Bucket. If you are interested, you can play with the scanner notebook here: https://observablehq.com/@jimpick/miner-report-maxmind-geolite2-lookups?collection=@jimpick/miner-report |
Beta Was this translation helpful? Give feedback.
-
|
Now that I’ve got location data in https://miner.report/ - I made a little notebook to explore where the miners are and organize them into geographic “Continent”-“Country” buckets. I also wanted to see how many miners are in multiple regions. I also added the ability to filter against ask/deal information. Nothing visual yet, but it’s fun to play with the data. I’m planning to output some of this so I can prepare aggregated stats by region. https://observablehq.com/@jimpick/miner-report-miner-to-region-mapper |
Beta Was this translation helpful? Give feedback.
-
|
Miner.Report update … I’m now computing aggregated statistics over time for the various “regions” I have bucketed miners into. Only power right now, but I’ll add some deal stats soon. => https://observablehq.com/@jimpick/miner-report-regional-stats?collection=@jimpick/miner-report |
Beta Was this translation helpful? Give feedback.
-
|
I am now breaking out the China-based miner stats by province - I wanted to do that since each province is bigger than many countries. Check it out here => https://observablehq.com/@jimpick/miner-report-regional-stats?region=AS-CN-XJ |
Beta Was this translation helpful? Give feedback.
-
|
Newest enhancement - Regional Deal Stats!! You can now see how many deals are happening in each region, how much data, how many providers, how many clients, etc. => https://observablehq.com/@jimpick/miner-report-regional-stats?region=NA-US |
Beta Was this translation helpful? Give feedback.
-
|
Pretty good. If it can be made into a website and updated every day, it will be a good product for users. |
Beta Was this translation helpful? Give feedback.
-
|
Newest addition to miner.report … a Zoomable “Sunburst” visualization! (Thanks to D3.js) See the network power, provider count, and deal data grouped by region over time. With animation support too! https://observablehq.com/@jimpick/miner-report-zoomable-sunburst?collection=@jimpick/miner-report
|
Beta Was this translation helpful? Give feedback.
-
|
I added a “Weekly Deals” notebook to https://miner.report … it shows some pretty nice growth in deals recently! https://observablehq.com/@jimpick/miner-report-published-deals-weekly?collection=@jimpick/miner-report
|
Beta Was this translation helpful? Give feedback.
-
|
The big renaming is done. The new name of the project is Provider.Quest and it can be found at https://provider.quest/ ... same data, just a new name! |
Beta Was this translation helpful? Give feedback.
-
|
I applied for another microgrant to "Add Historical Published Deal Data + Active/Slashed Deals" ... let's see if it's possible to get another one. filecoin-project/devgrants#397 I'd greatly appreciate any support! I didn't apply for a larger "open grant" as I'd prefer to just add small bits to this project over time when it makes sense and fits into my schedule. |
Beta Was this translation helpful? Give feedback.
-
|
Provider.Quest now has stats by individual country (plus US states and China provinces). https://observablehq.com/@jimpick/country-state-province-stats-provider-quest |
Beta Was this translation helpful? Give feedback.
-
Provider.Quest (formerly Miner.Report)
Provider.Quest is a project to collect analytics data from the Filecoin blockchain. It is also a collection of notebooks on ObservableHQ with interactive visualizations that use the data.
Access it at https://provider.quest/ (redirects to the Provider.Quest ObservableHQ Collection)
The project is live on ObservableHQ, with continuously updating deal data (daily and hourly), miner power and storage asks. The core data pipeline, powered by Apache Spark Structured Streaming, Textile Buckets and Estuary, is running 24/7 publishing new data multiple times per hour. The project is in an ideal state for new data sources, reports and visualizations to be added.
Beta Was this translation helpful? Give feedback.
All reactions