train 成功了，分享一些细节。 trained successfully! Share some details with guys. #25
Description
train成功了，分享一下环境：
- 系统环境：
- python==3.11.9
- transformers==4.33.0
- pytorch==2.2.0
- flash-attn==2.6.3
- ninja==1.11.1.1
- deepspeed==0.15.0
- wandb==0.17.8
- /glm-4-9b 目录下的
modeling_chatglm.py和tokenization_chatglm.py都已经替换 - 在 /ds_config/stage3.json 中设置
"stage3_prefetch_bucket_size": 15099494 - 在 /glm-4-9b/config.json 中设置
"seq_length": 131072
- general.json: (仅供测试)
{"messages":[{"role": "user", "content": "北京的天气怎么样？"}, {"role": "assistant", "content": "今天北京的天气晴朗，气温在25度左右。"}]}
{"messages":[{"role": "user", "content": "如何在Java中实现单例模式？"}, {"role": "assistant", "content": "你可以使用懒汉式或饿汉式来实现单例模式。以下是饿汉式的示例代码：\n```java\npublic class Singleton {\n private static final Singleton instance = new Singleton();\n private Singleton() {}\n public static Singleton getInstance() {\n return instance;\n }\n}\n```"}]}
{"messages":[{"role": "user", "content": "你知道中国的四大发明是什么吗？"}, {"role": "assistant", "content": "中国的四大发明是造纸术、指南针、火药和印刷术。"}]}
{"messages":[{"role": "user", "content": "有什么推荐的中文书籍吗？"}, {"role": "assistant", "content": "如果你喜欢历史类书籍，我推荐《资治通鉴》。如果你对文学感兴趣，可以看看《红楼梦》。"}]}
{"messages":[{"role": "user", "content": "What's the weather like in New York?"}, {"role": "assistant", "content": "The weather in New York is cloudy with a temperature around 18°C."}]}
{"messages":[{"role": "user", "content": "How can I implement a stack in Python?"}, {"role": "assistant", "content": "You can implement a stack using a list in Python. Here’s a simple example:\n```python\nstack = []\nstack.append(1)\nstack.append(2)\nprint(stack.pop()) # Outputs 2\n```"}]}
{"messages":[{"role": "user", "content": "Can you explain the concept of machine learning?"}, {"role": "assistant", "content": "Machine learning is a field of artificial intelligence where algorithms learn from data to make predictions or decisions without being explicitly programmed."}]}
{"messages":[{"role": "user", "content": "What's a good book to learn about AI?"}, {"role": "assistant", "content": "A great book to start with is 'Artificial Intelligence: A Modern Approach' by Stuart Russell and Peter Norvig. It covers a wide range of AI topics comprehensively."}]}-
tips:
-
/glm-4-9b 目录下的
modeling_chatglm.py和tokenization_chatglm.py都已经替换这一步不确定自己是否成功的小伙伴，可以在这两个代码中自己加一些注释，如果报错了就检查一下报错代码中有没有这些注释。
-