Skip to content

[Benchmark] Add support for MaRVL, xGQA and ALM-Bench#1504

Open
inigopm wants to merge 1 commit intoopen-compass:mainfrom
inigopm:pr-multilingual-benchmarks
Open

[Benchmark] Add support for MaRVL, xGQA and ALM-Bench#1504
inigopm wants to merge 1 commit intoopen-compass:mainfrom
inigopm:pr-multilingual-benchmarks

Conversation

@inigopm
Copy link
Copy Markdown
Contributor

@inigopm inigopm commented Apr 2, 2026 •
edited
Loading

Summary

  • add dataset implementations for MaRVL, xGQA, and ALM-Bench

Data note

  • this PR focuses on the benchmark implementations and registration split requested for review
  • the main open point is the preferred upstream data strategy for the large multilingual TSVs, especially for datasets whose images should not be mirrored casually because of size or licensing constraints
  • I can follow maintainers' preferred direction here, whether that is official-source local preparation, maintainer-managed hosting, or another existing VLMEvalKit pattern

@inigopm @inakiLakunza
[Benchmark] Add support for MaRVL, xGQA and ALM-Bench.
5972c40 
Register the three multilingual benchmarks and resolve generic config entries to their dataset-specific implementations so they can be evaluated through the standard VLMEvalKit flow.

Co-authored-by: Iñaki Lakunza <136484940+inakiLakunza@users.noreply.github.com>
Made-with: Cursor
@inigopm inigopm force-pushed the pr-multilingual-benchmarks branch from 9656947 to 5972c40 Compare April 2, 2026 10:06
@kennymckormick
Copy link
Copy Markdown
Member

kennymckormick commented Apr 6, 2026

Hi, @inigopm ,

Thanks for the contribution, I suggest that you can share the data files via an anonymous huggingface account, or any approaches you prefer. Then we can upload the data to maintainer managed hosting service.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@inigopm @kennymckormick