Replies: 1 comment 2 replies
-
🤖 你好,@278038551!我是Dosu,一个可以帮助你解决问题,回答疑问,甚至成为贡献者的机器人。我在这里为你提供帮助,直到有人类维护者赶来。让我们一起解决你的问题吧! 您的问题可能是由于在 try: ## 优先使用用户自定义的text_splitter
text_splitter_module = importlib.import_module('text_splitter')
TextSplitter = getattr(text_splitter_module, splitter_name)
except: ## 否则使用langchain的text_splitter
text_splitter_module = importlib.import_module('langchain.text_splitter')
TextSplitter = getattr(text_splitter_module, splitter_name) 如果在执行整个项目时在'text_splitter'模块中找不到自定义拆分器,可能是由于模块不在Python路径中。如果在调试和执行期间从不同的目录运行项目,可能会发生这种情况。 为了解决这个问题,您可以确保在运行项目时'text_splitter'模块在Python路径中。您可以通过在运行项目之前将包含'text_splitter'模块的目录添加到Python路径来实现这一点。这可以通过设置PYTHONPATH环境变量或在Python脚本中修改sys.path来完成。 如果问题仍然存在,您可能需要检查'kb_config'中的配置,以确保正确指定了自定义拆分器。 参考源码: 另外,关于'ChineseRecursiveTextSplitter'类和'_split_text_with_regex_from_end'函数的实现,我在提供的Langchain-Chatchat代码库上下文中并未找到'make_text_splitter'函数的实现。因此,我无法根据给定的上下文为'make_text_splitter'函数提供解释或代码。 参考源码: SourcesThis response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
向大神求助,新增custom_text_splitter.py,debug该文件,发现自定义的文档拆分方法生效了,但是运行整个项目,未按预期拆分,请问这是什么原因啊?
已配置kb_config,配置信息如下:
"CustomTextSplitter": {
"source": "huggingface", # 选择tiktoken则使用openai的方法
"tokenizer_name_or_path": "",
},
Beta Was this translation helpful? Give feedback.
All reactions