[run llama-13b] NameError: name 'cuda_setup' is not defined. Did you mean: 'CUDASetup'?

[s1530129650]

run python server.py --share --listen --chat --model llama-7b.

Everything is okay.

However, when run ```` python server.py --share --listen --chat --model llama-13b ```,

It would report an error: NameError: name 'cuda_setup' is not defined. Did you mean: 'CUDASetup'?

[s1530129650]

fixed by

pip uninstall bitsandbytes
pip install bitsandbytes

refer to bitsandbytes-foundation/bitsandbytes#134.

Another issue
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

[Ph0rk0z]

You will have to add #198 and set threshold probably between 1 and 1.5 and limit context to where you don't get NaN error or out of memory. For llama 13b I can pull this off, for pythia 12b I cannot.

[github-actions]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.