Search results not clickable when using file connector and specifying metadata link:

[dbro]

Using the file connector and the metadata as described here https://docs.danswer.dev/connectors/file , the links are not working.

In the html of the page, the value of the href property is an empty string, and not the value of the link specified. For comparison, the value specified for the file_display_name is working as expected.

Here is the test file I uploaded to Danswer:

#DANSWER_METADATA={"link": "https://docs.danswer.dev/connectors/file", "file_display_name": "Danswer can index files, too", "primary_owners": ["abc@123.xyz", "def@456.xyz"], "test_tag":"test_value"}
# File Connector

## Access knowledge from Local Files

How it works

The File Connector indexes user uploaded files. - Currently supports .txt, .pdf, .md and .mdx files - Can also upload a .zip containing these files - If there are other file types in the zip, the other file types are ignored - Optional metadata line that supports links, document owners, and time updated as metadata for Danswer’s retrieval and AI Answer

Adding Metadata

The metadata line should be placed at the very top of the file and can take one of two formats:

These screenshots show what appears in the search results, and in the page html. Notice the blue highlighted line has href="" instead of the expected href="https://docs.danswer.dev/connectors/file"

[dbro]

There might be an issue with an incorrect variable name, introduced with commit a4d5ac8

This line works and refers to the variable called file_metadata :
https://github.com/danswer-ai/danswer/blob/3f1cd1ad129683090d85a15f6208bfd5a9428100/backend/danswer/connectors/file/connector.py#L72

This line might not work, and it refers to the variable called metadata :
https://github.com/danswer-ai/danswer/blob/3f1cd1ad129683090d85a15f6208bfd5a9428100/backend/danswer/connectors/file/connector.py#L101

So perhaps changing line 101 to use file_metadata.get() instead of metadata.get() would fix it?

Note there are other references to metadata.get() that might need to be updated:
https://github.com/danswer-ai/danswer/blob/3f1cd1ad129683090d85a15f6208bfd5a9428100/backend/danswer/connectors/file/connector.py#L78
https://github.com/danswer-ai/danswer/blob/3f1cd1ad129683090d85a15f6208bfd5a9428100/backend/danswer/connectors/file/connector.py#L106
https://github.com/danswer-ai/danswer/blob/3f1cd1ad129683090d85a15f6208bfd5a9428100/backend/danswer/connectors/file/connector.py#L107

this line is probably ok (?)
https://github.com/danswer-ai/danswer/blob/3f1cd1ad129683090d85a15f6208bfd5a9428100/backend/danswer/connectors/file/connector.py#L135

[eojthebrave]

It looks like this was broken again by the recent refactoring of the file utility functions in #1449. That PR introduced the read_text_file function. And in backend/connectors/file/connector.py the function needs to be called with the ignore_danswer_metadata=False. Like this:

file_content_raw, file_metadata = read_text_file(file, encoding=encoding, ignore_danswer_metadata=False)

Otherwise it will simply bypass even trying to read the #DANSER_METADATA content from the file.

[mcandio]

Hey @eojthebrave, is it possible to reference the file internally? or is it mandatory to be an external link?
I mean, I want to open the file that I recently uploaded but I keep receiving a 404, do I need to set a specific path?

[eojthebrave]

@mcandio I'm not sure. I'm very new to this project. What do you mean by reference it internally?

[mcandio]

@mcandio I'm not sure. I'm very new to this project. What do you mean by reference it internally?

I mean, for example, if I create the .danswer_metadata.json like this:

[
    {
        "file_display_name": "filename",
        "filename": "filename.pdf",
        "link": "./filename.pdf"
    }
]

What should be the link to the internal path where the file is stored? does the background deployment or the postgres database creates internal links to host these files?

[SingTeng]

I am having issue of result not clickable when using file connector, may I know if this issue has been resolved? Or is there any temporary fix?

[github-actions]

This issue is stale because it has been open 75 days with no activity. Remove stale label or comment or this will be closed in 15 days.

[KaranSantra]

This issue still exists. The source is not clickable or taking me to the "link" I provided in the metadata json. While uploading as a .zip using the file connector.

I tested with uploading a two pdfs and the .onyx_metadata.json containing

[
{
"filename": "Tracking transactions discounts.pdf",
"link": "https://docs.example.com/PDF Guides.html",
"file_display_name": "Pdf Guides"
},
{
"filename": "Assigning discounts to items.pdf",
"link": "https://docs.example.com/assigning-gl-accounts.html",
"file_display_name": "Assigning Gl Accounts"
}
]

[ciaransweet]

I've also come across this issue:

    {
        "filename": "Makefile.txt",
        "link": "https://github.com/ecmwf/earthkit/blob/develop/Makefile",
        "file_display_name": "Makefile",
        "primary_owners": [
            "ECMWF"
        ]
    },

The link in the chat opens up the file in a modal, it doesn't direct me to the link specified in the .onyx_metadata.json file

[ciaransweet]

Hey @evan-onyx sorry for the direct ping!

I see you raised: #4552 which is available in v0.28.0-beta.1 which I'm currently using, I can recreate this behaviour with .zip file containing a markdown file and the .onyx_metadata.json file.

For example:

[
    {
        "filename": "README.md",
        "link": "https://github.com/ciaransweet/action-stations/blob/main/README.md",
        "file_display_name": "Action Stations README.md",
        "primary_owners": ["ciaran"]
    }
]

Is my understanding correct that link should be what I see on the little file prompts up top? I.E. for my metadata file, I should actually see a GitHub link to the README.md?

Perhaps something changed in the implementation but wasn't covered off in docs?

[evan-onyx]

Hey @ciaransweet Thanks for bringing this to our attention! I tried recreating this in the latest main (which should be similar to the code from 0.28.0-beta.1 as I don't think anything major has changed) but was unable to. Could you try unzipping and confirming that the metadata file is at the top level of the zip file (i.e. not inside a folder)? If you i.e. compress a folder containing the metadata file and the readme file, Onyx won't find the metadata because it expects it at the top level of the zip. In order to get the desired file structure, you can run the following on Unix-y systems:

zip my_zip_file.zip .onyx_metadata.json <first_file> <second file> <etc> and use -r if you want to add folders to the zip (only the .onyx_metadata.json needs to not be in a folder)