add performance statistics for image generation

[xufang-lisa]

No description provided.

[ilya-lavrenov]

Could you please provide example of such prints?
E.g. for SDXL and FLUX

[likholat]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

struct OPENVINO_GENAI_EXPORTS ImageGenerationPerfMetrics {
    float load_time; // model load time (includes reshape & read_model time)
    float generate_duration; // duration of method generate(...)

    MeanStdPair iteration_duration; // Mean-Std time of one generation iteration
    std::map<std::string, float> encoder_inference_duration; // inference durations for each encoder
    MeanStdPair unet_inference_duration; // inference duration for unet model, should be filled with zeros if we don't have unet
    MeanStdPair transformer_inference_duration; // inference duration for transformer model, should be filled with zeros if we don't have transformer
    float vae_encoder_inference_duration; // inference duration of vae_encoder model, should be filled with zeros if we don't use it
    float vae_decoder_inference_duration; // inference duration of  vae_decoder model
 
    bool m_evaluated = false;
 
    RawImageGenerationPerfMetrics raw_metrics;
};
 
struct OPENVINO_GENAI_EXPORTS RawImageGenerationPerfMetrics {
    std::vector<MicroSeconds> unet_inference_durations; // unet durations for each step
    std::vector<MicroSeconds> transformer_inference_durations; // transformer durations for each step
    std::vector<MicroSeconds> iteration_durations;  //  durations of each step
};

I'd also like to propose return ov::Tensor as a generate method output and add get_perfomance_metrics() method

[xufang-lisa]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

updated

[ilya-lavrenov]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

updated

could you please add benchmark for image generation similar to benchmark for GenAI LLM, which can print all these detailed statistic?
See https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/cpp/benchmark_genai/benchmark_genai.cpp#L54-L61

And provide here an output of such sample.

BTW, original samples should be kept as is and avoid printed timings.

[xufang-lisa]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

updated

could you please add benchmark for image generation similar to benchmark for GenAI LLM, which can print all these detailed statistic? See https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/cpp/benchmark_genai/benchmark_genai.cpp#L54-L61

And provide here an output of such sample.

BTW, original samples should be kept as is and avoid printed timings.

@ilya-lavrenov Do you mean to add support for image generation in benchmark_genai?

[ilya-lavrenov]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

updated

could you please add benchmark for image generation similar to benchmark for GenAI LLM, which can print all these detailed statistic? See https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/cpp/benchmark_genai/benchmark_genai.cpp#L54-L61
And provide here an output of such sample.
BTW, original samples should be kept as is and avoid printed timings.

@ilya-lavrenov Do you mean to add support for image generation in benchmark_genai?

no, create dedicated benchmark application for image generation, which is similar to VLM / LLM benchmarks