Fullchain.pem expired / force renewal doesn't renew

[tomtko]

So, just recently i've been getting curl certificate expired errors on my ubuntu 14.04 server, however it works just fine on any other server, including my local machine
curl: (60) SSL certificate problem: certificate has expired

A bit of investigating, I believe that the Intermediate has expired or is invalid, whynopadlock result shows that, along with /app/cert_status command showing that the fullchain.pem has expired, force renewing doesn't seem to fix the issue, it 100% renews the certificate however.

##### Certificate status #####
/etc/nginx/certs/domain/fullchain.pem: EXPIRED
Certificate was issued by R3
Certificate was valid until Jan  9 05:04:47 2022 GMT
Subject Name:
- domain
Certificate is used by the following domain(s):
- domain

---

Info: running acme-companion version v2.1.0-28-g1785bc5

version: '2'
services:
  nginx-proxy:
    image: nginxproxy/nginx-proxy:alpine
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - conf:/etc/nginx/conf.d
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - dhparam:/etc/nginx/dhparam
      - certs:/etc/nginx/certs:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro

  nginx-proxy-letsencrypt:
    image: nginxproxy/acme-companion
    container_name: nginx-proxy-letsencrypt
    volumes_from:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs:rw
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - DEFAULT_EMAIL=

  whoami:
    image: jwilder/whoami
    environment:
      - VIRTUAL_HOST=whoami.local

volumes:
  conf:
  vhost:
  html:
  dhparam:
  certs:
  acme:

---

rendered nginx configuration

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  10240;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}
daemon off;

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
  default $http_x_forwarded_port;
  ''      $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
  default off;
  https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent" '
                 '"$upstream_addr"';
access_log off;
                ssl_protocols TLSv1.2 TLSv1.3;
                ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
                ssl_prefer_server_ciphers off;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        server_tokens off;
        listen 80;
        access_log /var/log/nginx/access.log vhost;
        return 503;
}
server {
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        server_tokens off;
        listen 443 ssl http2;
        access_log /var/log/nginx/access.log vhost;
        return 503;
        ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;
        ssl_certificate /etc/nginx/certs/default.crt;
        ssl_certificate_key /etc/nginx/certs/default.key;
}
# domain
upstream domain {
        ## Can be connected with "nginx-proxy_default" network
        # domain_nginx_1
        server 172.19.0.5:80;
}
server {
        server_name domain;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        # Do not HTTPS redirect Let'sEncrypt ACME challenge
        location ^~ /.well-known/acme-challenge/ {
                auth_basic off;
                auth_request off;
                allow all;
                root /usr/share/nginx/html;
                try_files $uri =404;
                break;
        }
        location / {
                return 301 https://$host$request_uri;
        }
}
server {
        server_name domain;
        listen 443 ssl http2 ;
        access_log /var/log/nginx/access.log vhost;
        ssl_session_timeout 5m;
        ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;
        ssl_certificate /etc/nginx/certs/domain.crt;
        ssl_certificate_key /etc/nginx/certs/domain.key;
        ssl_dhparam /etc/nginx/certs/domain.dhparam.pem;
        ssl_stapling on;
        ssl_stapling_verify on;
        ssl_trusted_certificate /etc/nginx/certs/domain.chain.pem;
        add_header Strict-Transport-Security "max-age=31536000" always;
        include /etc/nginx/vhost.d/default;
        location / {
                proxy_pass http://domain;
        }
}
# whoami.local
upstream whoami.local {
        ## Can be connected with "nginx-proxy_default" network
        # nginx-proxy_whoami_1
        server 172.19.0.2:8000;
}
server {
        server_name whoami.local;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        include /etc/nginx/vhost.d/default;
        location / {
                proxy_pass http://whoami.local;
        }
}
server {
        server_name whoami.local;
        listen 443 ssl http2 ;
        access_log /var/log/nginx/access.log vhost;
        return 500;
        ssl_certificate /etc/nginx/certs/default.crt;
        ssl_certificate_key /etc/nginx/certs/default.key;
}

# configuration file /etc/nginx/vhost.d/default:
## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
    auth_basic off;
    auth_request off;
    allow all;
    root /usr/share/nginx/html;
    try_files $uri =404;
    break;
}
## End of configuration add by letsencrypt container

[buchdag]

Hi, isn't this caused by the DST Root CA X3 expiration ?

[gurumark]

I have the same issue and still could not figure out if it is related to this project or not.
docker exec letsencrypt /app/cert_status shows all certificates as expired however force renewal works all certificates are getting renewed. I use docker so the server being old should not be an issue.

[buchdag]

@gurumark @fluffyst nginxproxy/acme-companion:2.1.1 has been updated to use a newer version of acme.sh (2.8.8 -> 2.9.0), could you test again with this version of the container ?

[DatAres37]

Didn't fix it for me at least. I've got the same phenomenon, that cert_status shows all fullchains as expired (Certificate was valid until Mar 5 2022), but I don't have any other issues besides that. The certificates and the renewal seems to work as usual.

It doesn't seem to make a difference if I use ACME_PREFERRED_CHAIN on a container or not. I'm on a fully updated Ubuntu 20.04 machine with the latest images.

[ezequielbjm]

Did someone find the fix for this? I'm facing the same issue with the fullchain.pem file

[DatAres37]

The only way to show the fullchain as OK instead of expired is to manually edit the fullchain.pem and remove the last cert in the chain.

[buchdag]

Just to clarify: the cert_status function is a small utility that was hastily written a while ago, it's not meant to actually reflect the internal state of acme.sh in any way.

I know the preferred chain stuff has been an issue on the acme.sh side for a while, the more recent version 3.x might finally solve this but I'll have to check a few things before bumping to this version. It's fairly easy to locally build an acme-companion image with a different version of acme.sh, I can provide instructions for this if needed.

[seligmanns]

Since one year I'm running a mailcow-dockerized behind nginx-proxy.
Today I'm facing exactly this issue myself from one day to the other.
In my case IMAPS does not work anymore.
openssl connection to the server returns: certificate has expired
force renewal works but cert_status on my mailserver says that fullchain.pem has exired and:
"Certificate was valid until Jun 22 06:06:14 2022 GMT"
really strange

[x-yuri]

Hi, isn't this caused by the DST Root CA X3 expiration ?

I think that is so. I've created an issue on the Let's Encrypt forum. Follow the link to find the suggested solution.

I tried to find a solution myself, and the following is probably not needed anymore, but just in case, to extract the first certificate from a file:

$ awk '/BEGIN/{if (matched) exit; else matched=1}; {print}' < /etc/nginx/certs/example.com/chain.pem | less