Skip to content

Support query argument redaction in DM logging #11489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
michaelmdeng opened this issue Aug 15, 2024 · 5 comments · Fixed by #11709 or #11783 · May be fixed by #12070
Closed

Support query argument redaction in DM logging #11489

michaelmdeng opened this issue Aug 15, 2024 · 5 comments · Fixed by #11709 or #11783 · May be fixed by #12070
Labels
type/feature Issues about a new feature

Comments

@michaelmdeng
Copy link
Contributor

michaelmdeng commented Aug 15, 2024 •
edited
Loading

Is your feature request related to a problem?

When DM logs queries, either with debug logging for all queries, or when encountering errors with warning/error logging, it logs the entire text of the query with arguments. This risks logging sensitive data.

For example, many of the log lines in https://github.com/pingcap/tiflow/blob/master/dm/pkg/conn/baseconn.go will log (truncated) full queries and arguments

Describe the feature you'd like

Configuration, similar to tidb log redaction, that will redact sensitive arguments from queries when DM logs them.

Describe alternatives you've considered

Since they are logged at all possible log levels, no alternative but turning off logging entirely.

Teachability, Documentation, Adoption, Migration Strategy

N/A
@michaelmdeng michaelmdeng added the type/feature Issues about a new feature label Aug 15, 2024
@michaelmdeng michaelmdeng changed the title Support query redaction in DM Support query argument redaction in DM logging Aug 15, 2024
@db-will
Copy link
Contributor

db-will commented Oct 31, 2024

Hey @michaelmdeng , I want to follow up on this issue. Should we redact the SQL query, or is redacting the query arguments sufficient?

@db-will db-will mentioned this issue Nov 1, 2024
Merged
1 task
@ti-chi-bot ti-chi-bot bot closed this as completed in 5c7d683 Nov 5, 2024
@lance6716
Copy link
Contributor

There are still some places that forget to proactively use redact function.

@lance6716 lance6716 reopened this Nov 14, 2024
@db-will
Copy link
Contributor

db-will commented Nov 19, 2024

@lance6716 could you point out a few places that we might need to redact sensitive info? thanks

@lance6716
Copy link
Contributor

like

tiflow/dm/syncer/dbconn/db.go

Line 249 in 8e3b63a

zap.String("arguments", utils.TruncateInterface(args, -1)),

@db-will db-will mentioned this issue Nov 20, 2024
Merged
@db-will
Copy link
Contributor

db-will commented Nov 20, 2024

@lance6716 tried to find similar pattern as you presented, didn't find any other similar lines that print out query arguments. i create a update pr.

ti-chi-bot bot pushed a commit that referenced this issue Nov 21, 2024
@db-will
Redact query arguments in dm log (#11783)
f4dcfe4 
close #11489
mjonss pushed a commit to mjonss/tiflow that referenced this issue Nov 25, 2024
@db-will @mjonss
Redact query arguments in dm log (pingcap#11783)
3d99b81 
close pingcap#11489
@db-will db-will mentioned this issue Feb 23, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/feature Issues about a new feature
Projects
None yet
Milestone
No milestone
3 participants
@lance6716 @michaelmdeng @db-will