[Feature] Display the amount of processed data when running tests

[uatach]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

When executing the run or build command, dbt displays the amount of data processed to create tables, I think it would be nice it also displayed the amount of data processed when running tests

Describe alternatives you've considered

No response

Who will this benefit?

We had a situation that after writing some new models, a chart in a report tracking the amount of processed data in BigQuery showed a very sharp increase... we looked at the output from dbt and found that it was processing a lot of data to create some tables, we fixed that but the amount of data processed still increased more than expected, we learned that we had missed some tests that were scanning a lot of the same data and refactored the tests downstream to other models.

Are you interested in contributing this feature?

Yes but I don't know the codebase, no idea where I would start.

Anything else?

No response

[dbeatty10]

Thanks for reaching out @uatach !

I'm going to transfer this to the dbt-bigquery repo for further consideration since this is specific to BigQuery.

In the meantime, did you take a look at your target/run_results.json?

According to dbt-labs/dbt-core#2964 and dbt-labs/dbt-bigquery#559, the bytes processed should be available there.

But it seems like your ask is specific to the CLI output like this and you want PASS replaced with something like PASS (1.0 rows, 0 processed):

[uatach]

In the meantime, did you take a look at your target/run_results.json?
According to dbt-labs/dbt-core#2964 and dbt-labs/dbt-bigquery#559, the bytes processed should be available there.

Yes, indeed they are available there, thanks, didn't know about that.

But it seems like your ask is specific to the CLI output like this and you want PASS replaced with something like PASS (1.0 rows, 0 processed)

Yes, displaying it in the CLI output would be great.

[elyobo]

Agreed, I asked for this feature back in dbt-labs/dbt-bigquery#14 (comment) and it just occurred to me that it never showed up even though I thought that dbt-labs/dbt-bigquery#559 had added it! Didn't realise it was only in the JSON either.

Edit: JSON is probably better for some of my uses anyway, I have some dodgy CLI parsing somewhere that would be easier and more reliable with the JSON data.