Single-run Backfill Cloud Credits

[j-blackwell]

If I have a partitioned asset with 10,000 partitions and a backfill policy of single_run, how many credits will it cost to backfill this asset in a single run?

1 or 10,000?
The website is a bit unclear as there is only one operation that records 10,000 materialisations.

[gibsondan]

Each materialization uses up a credit, and each op uses up a credit - so a single-run backfill that materializes N assets will use N + 1 credits (N materializations and 1 op that does all N materializations)

[zyd14]

Agreed that this is quite the rough edge in the pricing. We were considering adopting partitions more for a number of use cases but will be reconsidering that plan as it would quickly blow out our enterprise plan. Seems like using partitions for data of any medium+ scale data is somewhat prohibitive in cost. Tens or hundreds of thousands of partitions is pretty small in the big data world; we have dozens of assets that would easily have tens or hundreds of thousands of partitions, the cost of modeling this with Dagster partitions far outweighs the value proposition.

[ion-elgreco]

I don't expect to ever use the enterprise features with this pricing model in place

[toddy86]

+1. I've mentioned this in the Slack threads before. It promotes anti-patterns to "work around" expensive backfill operations. I have a similar concern with the asset checks, which are apparently not being charged for (but by the sounds of it they will be).

Whilst Dagster Cloud is currently an affordable alternative to what we were using previously (MWAA), will all of these additional costs, it's quickly adding up and will very quickly exceed what we were paying. That said, we get a lot more value out of Dagster Cloud (branch deployment etc). So IMO, I don't mind paying the same or a bit more than e.g. MWAA. But if the price keeps escalating, it'll get to a point where it's not affordable for our small operation and we'll probably end up changing to OSS (which I am not a fan of, as I believe in supporting the providers of OSS).

[ion-elgreco]

@gibsondan why not just take the log to determine the pricing of a partitioned asset. log (partitions * 10)

So 10 partitions is 2 credits, 100 partitions 3 credits and so on. And you could say the starting price is always 2 credits for partitioned runs when it's above 2 partitions.. This would be way more friendly to do huge backfills, and ongoing partitioned materializations you will still pay extra for using the partitions which is a nice functionality.

[noahtf13]

Cannot agree more here. The pricing structure incentivizes you to not use the features as you should. We are very excited about auto-materilization, granular partitions, etc and now we are incentivized to use clunky async jobs and do local backfills because of this weird pricing structure.

I also can't imagine that Dagster's own costs scale with materializations but instead the actual compute time.

This is an amazing tool but pricing just makes it so frustrating.