New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can I feed this library raw counts instead of text files, and have it compute the Kneser Ney probabilities for me? #13

Open

GoogleCodeExporter opened this issue Jul 16, 2015 · 1 comment

Labels

auto-migrated Priority-Medium Type-Defect

GoogleCodeExporter commented Jul 16, 2015

If we have a very large corpus that I would like to take counts of in some 
distributed way, is there a way to give those raw counts to this code to build 
my model for me?

Original issue reported on code.google.com by [email protected] on 17 Jul 2013 at 7:27

The text was updated successfully, but these errors were encountered:

Author

GoogleCodeExporter commented Jul 16, 2015

The answer is "sort of". There is code in place to estimate Kneser Ney 
probabilities from a Google-ngram-formatted corpus (see 
https://groups.google.com/forum/#!topic/berkeleylm-discuss/G6Ta2YTsAA0). 
However, there may be some bugs. But please try running it, and seeing what 
happens. If it crashes, I'll have extra incentive to fix it.

Original comment by [email protected] on 17 Jul 2013 at 8:14

GoogleCodeExporter added Priority-Medium Type-Defect auto-migrated labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment