Containers can’t reach each other via FQDN through Caddy in Podman setup

[m0canu1]

I have a Rocky Linux host with Podman.
• The host has a private IP (let’s say 192.168.1.1).
• There is a Caddy container running with socket activation (like example4). This container is connected to two Podman networks: caddy-grafana and caddy-prometheus.
• Grafana runs in its own container and is attached to caddy-grafana.
• Prometheus runs in its own container and is attached to caddy-prometheus.
• In the Caddyfile, I configured two domains:
• grafana.example.com → Grafana
• prometheus.example.com → Prometheus

From outside the host everything works correctly. When I open grafana.example.com or prometheus.example.com in a browser, I reach the right service.

The issue: inside the Grafana container, if I try to connect to prometheus.example.com, DNS resolves it to the host IP (192.168.1.1), but the connection doesn’t work.

How can I make containers reach each other through their FQDNs, the same way it works from outside?

[m0canu1]

Sorry, I wanted to put this "issue" on another repo (podman-caddy-socket-activation). But could also be useful here for someone else

[eriksjolund]

There are two ways the grafana container can connect to caddy, either

• directly on the custom network caddy-grafana (alternative 1)
• via the host's main network interface (alternative 2).

Alternative 1:

Add

NetworkAlias=prometheus.example.com

to the file caddy.container under the [Container] section.

Alternative 2:

Add

AddHost=prometheus.example.com:host-gateway

to the file grafana.container under the [Container] section.

For details, see the section Access proxy from internet, host and custom network

discussion

To use alternative 1, caddy needs to create listening sockets on the network caddy-grafana. The grafana container could then connect to caddy directly on the same network.

Currently caddy in Example 4 only creates socket TCP/80 on the custom network, but my plan is to change this so that it instead creates TCP/80, TCP/443, UDP/443 to be able to support HTTP/3.

[m0canu1]

I modified the Caddyfile adding those bind values and now seems to create a socket on port TCP/443. Now I get another kind of error but I don't know if it's related to connectivity issues or something else:
response from prometheus couldn't be parsed. it is non-json: ReadObject: expect { or , or } or n, but found , error found in #0 byte of ...||..., bigger context ...||... - There was an error returned querying the Prometheus API.

These are the first lines of my Caddyfile now:

http:// {
	bind fd/3 {
		protocols h1
	}
	bind :80
	redir https://{host}{uri}
	log
}

https:// {
	bind :443
	log
}