podman-compose pods hang when non unicode data is written in stdout/stderr.

[astierw-bull]

Describe the bug

We have a service that manages remote hardware, when errors occurs on remote hardware, we could receive binary data instead of text, and we may write this binary data in our logs redirected in the stderr of our pod.
When this happen, the podman-compose stops polling the incoming pipes, this blocks our pod in a write system call because the pipe is full.

Here is a strace sequence of podman-compose when the binary data is read by podman-compose:

[pid 1486179] epoll_wait(8, [{events=EPOLLIN, data={u32=13, u64=140003048947725}}], 5, -1) = 1
[pid 1486179] read(13, "\314\0\0\0\0\2\244\16\20\214\2A\16\30\206\3D\16 U\16\30G\306\16\20B\314\16\10\0\0"..., 262144) = 16384
[pid 1486179] epoll_wait(8, [{events=EPOLLIN, data={u32=13, u64=140003048947725}}], 5, 0) = 1
[pid 1486179] epoll_ctl(8, EPOLL_CTL_DEL, 13, 0x7ffc7141422c) = 0
[pid 1486179] epoll_wait(8, [{events=EPOLLIN, data={u32=15, u64=140003048947727}}], 4, -1) = 1
[pid 1486179] read(15, "01/08/2026 03:36:25.122 | HAL-DI"..., 262144) = 202
[pid 1486179] epoll_wait(8, [], 4, 0) = 0

This EPOLL_CTL_DEL is probably induced by an UnicodeDecodeError exception.

To Reproduce
To reproduce the problem with a minimal code, we created a short python script that creates a random binary data char array and writes it in stderr:

# cat bin_log.py
import time
import sys
import random

while True:
    print("Before binary", file=sys.stderr)
    s=''
    for i in range (50000):
        s+=chr(random.randrange(0,65535))
    print(s, file=sys.stderr)
    print("After binary", file=sys.stderr)
    time.sleep(1)

we embed that in an image:

# cat Dockerfile
FROM registry.access.redhat.com/ubi9-minimal:9.6

COPY bin_log.py .

RUN microdnf -y install python39

CMD ["python3", "bin_log.py"]

then we launched it using podman-compse with the following docker-compose file:

# cat docker-compose.yml
version: "3.3"
services:
  reproducer:
    image: "localhost/reproducer_podman_compose:latest"

when started with podman-compose up, podman-compose hangs and the phyton program is locked in write syscall.

Expected behavior
Pod does not hang as it was doing in podman-compose 1.0.6

Actual behavior
podman-compse up hangs the pod and when interrupting using ^C we receives the error:

File "/usr/lib/python3.9/site-packages/podman_compose.py", line 1608, in _format_stream
_formatted_print_without_nl(part.decode())
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 73726-73727: unexpected end of data

NB: this error mentions unicode error, but we send random binary data, not unicode char, we think that podman-compose should handle (survive) to any kind of data sent in stderr/stdout of the pod.

Output

# podman-compose --version
podman-compose version 1.5.0
podman version 5.4.0

...
$ podman-compose up
(...)
^CERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-15' coro=<Podman._format_stream() done, defined at /usr/lib/python3.9/site-packages/podman_compose.py:1581> exception=UnicodeDecodeError(
(...)
', 73726, 73728, 'unexpected end of data')>
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/podman_compose.py", line 1608, in _format_stream
    _formatted_print_without_nl(part.decode())
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 73726-73727: unexpected end of data

Environment:

• OS: Linux (RHEL96)
• podman version: 5.4.0
• podman compose version: 1.5.0

Additional context

To fix podman-compose, we create this patch:

# diff /usr/lib/python3.9/site-packages/podman_compose.py /usr/lib/python3.9/site-packages/podman_compose.py.ori
1605c1605
<                     _formatted_print_with_nl(part.decode(errors='ignore'))
---
>                     _formatted_print_with_nl(part.decode())
1608c1608
<                     _formatted_print_without_nl(part.decode(errors='ignore'))
---
>                     _formatted_print_without_nl(part.decode())

This issue may somehow be linked to issue #1308

[SckyzO]

I can confirm that this bug is real and reproducible.

In our case, services sometimes write non-UTF-8 / binary data to stdout or stderr (for example when interacting with hardware or returning raw error payloads). When this happens, podman-compose stops consuming the pipe due to a UnicodeDecodeError, which eventually causes the container to block once the pipe buffer is full.

I tested the following minimal patch in podman_compose.py, and it completely fixes the issue on our side:

# diff /usr/lib/python3.9/site-packages/podman_compose.py /usr/lib/python3.9/site-packages/podman_compose.py.ori
1605c1605
<                     _formatted_print_with_nl(part.decode())
---
>                     _formatted_print_with_nl(part.decode(errors='ignore'))
1608c1608
<                     _formatted_print_without_nl(part.decode())
---
>                     _formatted_print_without_nl(part.decode(errors='ignore'))

With this change:

podman-compose no longer crashes or stops reading stdout/stderr

The pod does not hang

Binary output is safely ignored instead of breaking log processing

This behavior is consistent with what other tools do when consuming arbitrary process output. From a robustness standpoint, assuming UTF-8 text on stdout/stderr is unsafe.

Hopefully this helps confirm the root cause and provides a straightforward fix.

[whym]

NB: this error mentions unicode error, but we send random binary data, not unicode char, we think that podman-compose should handle (survive) to any kind of data sent in stderr/stdout of the pod.

I think decoding is there mainly for log prefixes. A log line is prefixed with [service_name] after a new line is detected. If we do the detection with a simple byte match without decoding, I expect there will be false positives within some multi-byte characters.

It's true that we should and can recover from decode errors. I'm not sure if we can avoid losing data, though.