longlora-paddle #9939
longlora-paddle #9939
Conversation
|
Thanks for your contribution! |
|
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #9939 +/- ##
============================================
- Coverage 50.91% 17.38% -33.53%
============================================
Files 751 752 +1
Lines 120074 120288 +214
============================================
- Hits 61131 20916 -40215
- Misses 58943 99372 +40429 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| @@ -213,6 +213,111 @@ def parallel_matmul(x: Tensor, y: Tensor, transpose_y=False, tensor_parallel_out | |||
| return logits | |||
|
|
|||
|
|
|||
| def shift(qkv, bsz, q_len, group_size, num_heads, head_dim): | |||
There was a problem hiding this comment.
不要把longlora这部分写到llama的组网里面，可以考虑写到paddlenlp/transformers/longlora.py中，如果使用longlora那么调用replace函数，现在llama组网已经太复杂了，不要再往这里写了。https://github.com/dvlab-research/LongLoRA/blob/main/llama_attn_replace.py#L460
|
|
||
| def run_predictor(self, config_params=None): | ||
| if config_params is None: | ||
| config_params = {} |
There was a problem hiding this comment.
test longlora是不是可以直接写一个测试longlora函数是否能够跑通并且结果符合预期的测试就好，本身已经有一个longlora训练的测试了
|
This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动，被标记为stale。 |
PR types
New features
PR changes
Models
Description
Paddle implementation of LongLoRA