How to avoid of chat main example stops abruptly? #7180
Unanswered
Zapotecatl
asked this question in
Q&A
Replies: 1 comment 1 reply
-
If I see it right, you're setting -n (--n-predict) to 64 - which means the server is supposed to stop generating after 64 tokens. You can set it to a larger value (the same problem, just different threshold), or -1: this is IIRC unlimited, of course, there's risk with unlimited - some models for some prompts can start babling nonsense or even loop. |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi,
In the main example there is a part where if an error occurs the chat ends:
I don't understand in which cases the error occurs and how to avoid it. The parameters I use in my command are this:
-m C:\\model\\llama2_7b_chat_uncensored.Q4_K_M.gguf -f C:\\model\\chat-with-bob.txt -c 2048 -b 2048 -n 64 --keep -1 --temp 0.9 -- repeat_penalty 1.1 -i -r "User:" --log-disable
At the moment I am using a work around in which if the error occurs within an external while(true) the chat is launched again and it gives the impression that the chat continues and the user does not perceive that the program ended abruptly.
So, I'm wondering if Is there a solution to ensure that the error never occurs to avoid my ugly work around?
Thanks for the help!
Beta Was this translation helpful? Give feedback.
All reactions