Need proper handling for invalid character for CDATA field

[firodj]

When output tests contains colored output log that may include escape character (0x1b) on test output report, the escape character is not treated well by encoding/xml package for CDATA section, so it will generate invalid XML document (as spec), that may result the test report being unreadable by the CI (in my case GitLab CI).

Related issue: golang/go#53728

[jstemmer]

Thanks. It sounds like we should at least strip the invalid characters from the CDATA section. However, since ANSI escape sequences also contain regular characters we'd probably want to strip the entire ANSI sequence rather than just the (invalid) control characters.

[emtammaru]

I just ran into the same issue with Jenkins junit parser:

org.dom4j.DocumentException: Error on line 5 of document  : An invalid XML character (Unicode: 0x1b) was found in the CDATA section.

[janisz]

Same here, I've tried to update to v2 stackrox/stackrox#2586 and prow does not parse XML as it's not valid.

error on line 2291 at column 25: Input is not proper UTF-8, indicate encoding !
Bytes: 0x00 0x63 0x32 0x68

[jstemmer]

@janisz Where do these invalid bytes come from, this doesn't look like an ansi escape character. Any concerns if they are just dropped completely from the output?

[janisz]

This bytes came from key that is logged. Key is generated as prefix + separator + id where separator is []byte("\x00") which is a control character. I found a workaround of this and print the key with %q instead of %s. From my perspective it's save to ignore them or escape.
https://github.com/stackrox/stackrox/blob/1c51bfebc23b2748bc103a583b5460611dfcad53/pkg/dbhelper/db.go#L10

Minimal example:

package main

import "testing"

func TestEscape(_ *testing.T) {
	println("Test '\x00'")
}

 $ go test ./... -v  2>&1 | go-junit-report -set-exit-code | file -bi -
+ Actual  : application/octet-stream; charset=binary
- Expected: text/xml; charset=us-ascii

[janisz]

Created a PR to drop control characters from output #140

[jstemmer]

Leaving this issue open for now, because it would be nice if we could also detect ANSI escape sequences and remove them from the output.