Skip to content

esgpull doesn't mark downloaded files as done #70

@AtefBN

Description

@AtefBN

Query in question:

<dae248>
└── tags:           high_criteria_latest_prio23_source
    distrib:        True
    latest:         True
    replica:        None
    retracted:      False
    activity_id:    CFMIP, CMIP
    experiment_id:  amip, amip-piForcing
    grid_label:     gn
    institution_id: NASA-GISS
    project:        CMIP6
    source_id:      GISS-E2-1-G
    table_id:       E3hrPt
    variable_id:    cltcalipso
    variant_label:  r1i1p1f1
    files:          5.1 GiB / 8.0 GiB [4/6]

Runs into some issues moving the tmp done file to the designated targed directory.

[2025-05-09 12:14:53]  ERROR     esgpull
File rename error, shutil.copyfile was used instead.
For large files, download times might be impacted.
To address this issue, you may consider setting your `tmp` directory to the same filesystem as your `data` directory:

$ esgpull config path.tmp <some/path/on/data/filesystem>

[2025-05-09 12:14:53]  INFO      esgpull
[b blue]<7e999d>[/] · [green]1.4 GiB[/] · [red]4.6 MiB/s[/] · [blue]dpesgf03.nccs.nasa.gov[/]

[2025-05-09 12:15:42]  DEBUG     httpcore.http11
receive_response_body.complete

[2025-05-09 12:15:42]  DEBUG     httpcore.http11
response_closed.started

[2025-05-09 12:15:42]  DEBUG     httpcore.http11
response_closed.complete

[2025-05-09 12:15:42]  ERROR     esgpull
[Errno 18] Invalid cross-device link: '/mnt/scality/tools/.esgpull/tmp/69dc95a0f214667d21c8406072394f34f453d2c4.done' -> '/mnt/scality/WCRP/CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/amip/r1i1p1f1/E3hrPt/cltcalipso/gn/v20200102/cltcalipso_E3hrPt_GISS-E2-1-G_amip_r1i1p1f1_gn_198901010000-199812312100.nc'

But despite having successfully moved the file, the transfers are still marked as error in the db.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions