podman commit is very slower with large container diff？

[lddlww]

we create podman container for run llm， and download some models from huggingface， this result diff is very large， then we run podman commit --log-level debug -f oci xxx xxx:v1, but it run slower and paused by follow codes:
1:

podman/vendor/github.com/containers/buildah/image.go

Line 1065 in 2e738b9

rc, err = i.store.Diff("", layerID, diffOptions)

2:

podman/vendor/github.com/containers/buildah/image.go

Line 1110 in 2e738b9

size, err := io.Copy(writer, rc)

and we found srcHasher take a lot of time，i wanna to kown , how can we improve commit performance?

thanks in advantage

[baude]

@mheon @nalind care to comment ?

[nalind]

The image format requires a digest of the contents of the layers, and that's generated at commit-time for layers that aren't inherited from base images. The amount of processing required to determine what should be included in the layer is affected by which storage driver is being used, but without podman info output, I don't know if there's anything to suggest there.

[lddlww]

@nalind podman info is:

host:
  arch: amd64
  buildahVersion: 1.41.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.66
    systemPercent: 0.13
    userPercent: 0.22
  cpus: 56
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: file
  freeLocks: 2047
  hostname: 10-58-3-21
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1074
      size: 1
    - container_id: 1
      host_id: 1541792
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1074
      size: 1
    - container_id: 1
      host_id: 1541792
      size: 65536
  kernel: 6.8.0-63-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 127952515072
  memTotal: 540894203904
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.22
      commit: 4de19b63a85efd9ea8503452179c371181750130
      rundir: /run/user/1074/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1074/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 8262512640
  swapTotal: 8589930496
  uptime: 2443h 8m 9.00s (Approximately 101.79 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  10.58.3.21:
    Blocked: false
    Insecure: true
    Location: 10.58.3.21
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: 10.58.3.21
    PullFromMirror: ""
  docker.repo.kpy.cloud:
    Blocked: false
    Insecure: false
    Location: docker.repo.kpy.cloud
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.repo.kpy.cloud
    PullFromMirror: ""
store:
  configFile: /opt/containers/group/5g2oa5pt/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /opt/containers/pub_img
    overlay.imagestore: /opt/containers/pub_img
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.13-1_amd64
      Version: |-
        fusermount3 version: 3.14.0
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
  graphRoot: /opt/containers/group/5g2oa5pt/5g2oa5pt-root
  graphRootAllocated: 53619982336
  graphRootUsed: 13967409152
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 66
  runRoot: /run/user/1074/containers
  transientStore: false
  volumePath: /opt/containers/group/5g2oa5pt/5g2oa5pt-root/volumes
version:
  APIVersion: 5.6.1
  Built: 1757932221
  BuiltTime: Mon Sep 15 18:30:21 2025
  GitCommit: ""
  GoVersion: go1.24.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.1

[lddlww]

@nalind reference to

podman/vendor/github.com/containers/buildah/image.go

Line 1095 in 2e738b9

writeCloser, err := archive.CompressStream(multiWriter, i.compression)

how can i reset i.compression parameter with podman commit? or in storage.conf?

[nalind]

If you can switch from using fuse-overlayfs to the kernel's overlay filesystem, reading the data should require less CPU.
Compression is usually not used unless the destination requires it, which shouldn't be the case when committing to local storage.

[lddlww]

@nalind thanks for reply， and i found io.Copy can not set buf size manually，default buf size is 32k which could lead to slower copy for large diff , if podman commit and podman push could pass buf size parameter，that's would be great

[lddlww]

i am confused about for follow two code:
1:

podman/vendor/github.com/containers/buildah/image.go

Line 1110 in 2e738b9

size, err := io.Copy(writer, rc)

2: copy blob

podman/vendor/go.podman.io/image/v5/storage/storage_dest.go

Line 285 in 6fe36e2

decompressed, err := archive.DecompressStream(stream)

podman/vendor/go.podman.io/image/v5/storage/storage_dest.go

Line 295 in 6fe36e2

_, err = io.Copy(diffID.Hash(), decompressed)

if first one exe io.Copy why second code used Copy too？does it instead of mv?