support nvidia gpu mig resources

[pisko1979]

Hi all

thanks for time to make tools, great job!
we are using your tool to track gpu resources that are allocated to pods/nodes. it works great with standard
nvidia.com/gpu resource description.

we have nVidia H100 GPU and configured it to use MIG partition so we can run multiple pods on one gpu. it works great.
but after change, noticed that kubectl-view-allocation doesnt "find new virtual gpus" as they are registered on node as new Capacity nvidia.com/mig-xxx

this is describe of node having one H100 and applied mixed, all-balanced mig profile.

nvidia.com/gpu: 0
nvidia.com/mig-1g.12gb: 2
nvidia.com/mig-2g.24gb: 1
nvidia.com/mig-3g.47gb: 1

as there are no gpu (nvidia.com/gpu count 0) kubectl-view-allocation doesnt even list node that have GPU in it.
please check is there way to incude also nvidia.com/mig-*?

thanks
regards

[davidB]

Hi,
The tool lists every resource by default (no hardcoded filter or search), so if the node and pod have declared resources, they should be visible.

Can you share a sample (the "describe" yaml) of a node and pod (you can remove or ofuscate useless info (for this bug) like name, labels, volumes, events, ...)?

If no pod is allocating the "custom" resource ( via request or limit), then the resource may not be listed.

[pisko1979]

Hi David

thanks for your time.
as you can see from node describe, there is no nvidia.com/gpu, only nvidia.com/mig-xx.xx resource available. it is normal for MIG use case.

describe node (grep nvidia.com)
  nvidia.com/gpu:                 0
  nvidia.com/mig-1g.12gb:         2
  nvidia.com/mig-2g.24gb:         1
  nvidia.com/mig-3g.47gb:         1

pod is using gpu resource but not requesting nvidia.com/gpu as there is no availables gpu (just mig resources).

kubectl -n vllm describe deployment vllm |grep -i Limits: -A 4
    Limits:
      cpu:                     8
      memory:                  24Gi
      nvidia.com/mig-2g.24gb:  1

thanks

[davidB]

Can you share the full sections (grep could extract the info from labels, ...)?

kubectl describe node ...

Capacity:
  cpu:                2
  ephemeral-storage:  104548332Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8130832Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  101704617290
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             8130832Ki
  pods:               110
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                200m (10%)  0 (0%)
  memory             140Mi (1%)  170Mi (2%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)

What is the output of kubectl view-allocations -r nvidia?

In your pod do you declare the GPU in "request" also, or "limit" only?