Is there any plan to include offline-evaluation functionality?

[ita9naiwa]

Hi. I'm Hyunsung Lee and I'm using Implicit library for my project(and this library is very awesome!)

For now, there is no evaluation functions in this library so I am using my own evaluation functions evals such as Recall, Precision(thus also F-score), NDCG and (maybe) other metrics (mainly brought and slightly modified from @dawenl 's cofactor.

I can add these evaluation functions to this awesome library and it will be helpful for people who use this.
However, one problem is that I have no experience in programming with CUDA/C++ but only numerical python.

so, I'd like to ask these two question.

1. Is there any plan to include offline-evaluation for recommenders?
2. If not, can I add some evaluation metrics using Numpy/Scipy and Joblib(or other threading techniques)?

Sorry for that there's similar issue #30 but there are no new comments and seem no PR related to this for almost a year therefore I created new issue.

Thanks.

[benfred]

Thanks!

I have an initial version of calculating P@K here: https://github.com/benfred/implicit/compare/eval . I'm planning on extending this to MAP@K and NDCG@K and maybe MRR in the next couple of days.

I'm using cython/openmp to parallelize this - but with the GIL the benefits of doings this are pretty limited. Most of the time is spend in the 'model.recommend' call which requires the GIL. The matrix factorization based models see a modest speedup (around 4-5x on a 8 core system), but the rest of the models seem to run faster when using a single thread.

It would be much more efficient to let each model calculate recommendations for all users in parallel ( #72) and then score them here - but I thought it might make sense to get a slower version out first that requires less API changes.

[ita9naiwa]

Wow!
You created evaluation functions https://github.com/benfred/implicit/blob/master/implicit/evaluation.pyx
Thanks!

[evfro]

As a matter of providing more options, a comprehensive evaluation can be performed with the Polara framework. It has a straightforward support for implicit library (disclaimer: I'm the author).

See examples here:
https://github.com/Evfro/polara/blob/master/examples/Warm-start%20and%20standard%20scenarios.ipynb

[benfred]

@evfro: awesome! thanks for adding support. will check out soon