Skip to content

Conversation

twz123
Copy link
Member

@twz123 twz123 commented Sep 17, 2025

Sometimes, random log statements may appear on stderr, which can break the parsing. Try to parse stdout first, then fallback to stderr. Error out if none of both was a JSON document.

An example log line on stderr looks like this:

W0917 07:11:23.255691 3096 logging.go:55] [core] [Channel https://github.com//pull/1 SubChannel https://github.com//pull/2]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"

@kke
Copy link
Contributor

kke commented Sep 22, 2025

Would it be a problem to do it on all versions?

It's also possible to get separate streams with something like:

var stdoutbuf, stderrbuf bytes.Buffer
cmd, err := h.ExecStreams("command", nil, &stdoutbuf, &stderrbuf, exec.Sudo(h)) // first nil is stdin
if err != nil {
  return fmt.Errorf("failed to create command: %w", err)
}
if err := cmd.Wait(); err != nil {
  return fmt.Errorf("command failed: %w", err)
}
// parse stdout, maybe log stderr

@kke kke added the bug Something isn't working label Sep 22, 2025
@twz123
Copy link
Member Author

twz123 commented Sep 22, 2025

Would it be a problem to do it on all versions?

It's also possible to get separate streams with something like:

var stdoutbuf, stderrbuf bytes.Buffer
cmd, err := h.ExecStreams("command", nil, &stdoutbuf, &stderrbuf, exec.Sudo(h)) // first nil is stdin
if err != nil {
  return fmt.Errorf("failed to create command: %w", err)
}
if err := cmd.Wait(); err != nil {
  return fmt.Errorf("command failed: %w", err)
}
// parse stdout, maybe log stderr

Oh, that was the method I was looking for 😅

@twz123 twz123 marked this pull request as draft September 26, 2025 06:56
@laghoule
Copy link

I'm currently experiencing it:

INFO ==> Running phase: Gather k0s facts 
INFO [ssh] ctrl-01:22: found existing configuration 
INFO [ssh] ctrl-01:22: is running k0s controller+worker version v1.34.1+k0s.0 
WARN [ssh] ctrl-01:22: the controller+worker node will not schedule regular workloads without toleration for node-role.kubernetes.io/master:NoSchedule unless 'noTaints: true' is set 
INFO [ssh] ctrl-01:22: listing etcd members       
INFO * Running clean-up for phase: Acquire exclusive host lock 
INFO ==> Apply failed                    
FATA apply failed - log file saved to /home/XXX/.cache/k0sctl/k0sctl.log: failed to decode etcd member-list output: invalid character 'W' looking for beginning of value 

@twz123 twz123 linked an issue Oct 14, 2025 that may be closed by this pull request
4 tasks
Sometimes, random log statements may appear on stderr, which can break
the parsing. Try to parse stdout first, then fallback to stderr. Error
out if none of both was a JSON document.

An example log line on stderr looks like this:

    W0917 07:11:23.255691    3096 logging.go:55] [core] [Channel k0sproject#1 SubChannel k0sproject#2]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled"

Signed-off-by: Tom Wieczorek <twieczorek@mirantis.com>
@twz123 twz123 force-pushed the etcd-member-log-stderr branch from 0185b51 to e620285 Compare October 14, 2025 08:27
@twz123 twz123 changed the title Capture only stdout of k0s etcd member-list if possible Look at stdout and stderr of k0s etcd member-list separately Oct 14, 2025
@twz123 twz123 marked this pull request as ready for review October 14, 2025 09:59
result map[string]any
errs []error
)
for _, output := range [][]byte{stdout.Bytes(), stderr.Bytes()} {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a bit overkill to try parsing stderr for json, but sure, why not

@kke kke merged commit ebf27f5 into k0sproject:main Oct 14, 2025
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performing reset, "Failed to decode etcd member-list output"

3 participants