fix nltk punkt

[rkuo-danswer]

Description

Fixes https://linear.app/danswer/issue/DAN-1985/nltk-punkt-deprecated

related to nltk/nltk#3293

2025-05-19 08:53:43.267	
**********************************************************************

2025-05-19 08:53:43.267	
    - '/usr/local/lib/nltk_data'
2025-05-19 08:53:43.267	
    - '/usr/lib/nltk_data'
2025-05-19 08:53:43.267	
    - '/usr/local/share/nltk_data'
2025-05-19 08:53:43.267	
    - '/usr/share/nltk_data'
2025-05-19 08:53:43.267	
    - '/usr/local/lib/nltk_data'
2025-05-19 08:53:43.267	
    - '/usr/local/share/nltk_data'
2025-05-19 08:53:43.267	
    - '/usr/local/nltk_data'
2025-05-19 08:53:43.267	
    - '/root/nltk_data'
2025-05-19 08:53:43.267	
  Searched in:
2025-05-19 08:53:43.267	

2025-05-19 08:53:43.267	
  Attempted to load tokenizers/punkt_tab/english/
2025-05-19 08:53:43.267	

2025-05-19 08:53:43.267	
  For more information see: https://www.nltk.org/data.html
2025-05-19 08:53:43.267	
  
2025-05-19 08:53:43.267	
  >>> nltk.download('punkt_tab')
2025-05-19 08:53:43.267	
  >>> import nltk
2025-05-19 08:53:43.267	

2025-05-19 08:53:43.267	
  Please use the NLTK Downloader to obtain the resource:
2025-05-19 08:53:43.267	
  Resource punkt_tab not found.

How Has This Been Tested?

[Describe the tests you ran to verify your changes]

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 19, 2025 5:08pm

[greptile-apps]

PR Summary

Updates the NLTK Punkt tokenizer download command in /backend/Dockerfile to use 'punkt_tab' instead of 'punkt' due to deprecation of the original tokenizer.

• Changed RUN python -c "import nltk; nltk.download('punkt')" to use punkt_tab in /backend/Dockerfile to address NLTK issue #3293
• Ensures tokenization functionality remains intact by using the recommended replacement tokenizer

_{1 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile}