Skip to content

Commit aa31b0b

Browse files
authored
starcoder : add special tokens for fill-in-the-middle task (leejet#269)
1 parent 80be908 commit aa31b0b

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

examples/starcoder/main.cpp

+4
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,10 @@ bool starcoder_model_load(const std::string & fname, starcoder_model & model, gp
146146
"<|user|>",
147147
"<|assistant|>",
148148
"<|end|>",
149+
"<fim-prefix>",
150+
"<fim-middle>",
151+
"<fim-suffix>",
152+
"<fim-pad>"
149153
}) {
150154
if (vocab.token_to_id.find(token) != vocab.token_to_id.end()) {
151155
vocab.add_special_token(token);

0 commit comments

Comments
 (0)