Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data type error reported during vllm inference #3347

Closed
1 task done
zhangapeng opened this issue Apr 19, 2024 · 1 comment
Closed
1 task done

Data type error reported during vllm inference #3347

zhangapeng opened this issue Apr 19, 2024 · 1 comment
Labels
solved This problem has been already solved

Comments

@zhangapeng
Copy link

Reminder

  • I have read the README and searched the existing issues.

Reproduction

CUDA_VISIBLE_DEVICES=0 API_PORT=8000 python src/api_demo.py
--model_name_or_path /home/Baichuan2/Baichuan2-7B-Chat
--template baichuan2
--finetuning_type lora
--infer_backend vllm

When running on the v100 machine, an error occurs that BFfloat is not supported. Solution: Add parameter dtype='float16' in line 36 of src/src/llmtuner/chat/vllm_engine.py

Expected behavior

No response

System Info

No response

Others

No response
@hiyouga hiyouga added the pending This problem is yet to be addressed label Apr 19, 2024
@hiyouga
Copy link
Owner

hiyouga commented Apr 23, 2024

fixed

@hiyouga hiyouga added solved This problem has been already solved and removed pending This problem is yet to be addressed labels Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
solved This problem has been already solved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hiyouga @zhangapeng