Skip to content

fix: ignore comment lines at end of file in CSV parser #396

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

fix: ignore comment lines at end of file in CSV parser #396

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
193c231
test: add test for ignoring comments at end of file
DCjanus Apr 27, 2025
7b2be38
test: update comment handling test to ignore EOF comments
DCjanus Apr 27, 2025
f2cba42
fix: skip last line in comment
DCjanus Apr 27, 2025
095e842
fix: improve comment handling in CSV reader
DCjanus Apr 27, 2025
32d3649
test: add additional test for ignoring comments at end of file
DCjanus Apr 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion csv-core/src/reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -752,7 +752,10 @@ impl Reader {
// parsing a new record, then we should sink into the final state
// and never move from there. (pro-tip: the start state doubles as
// the final state!)
if state >= self.dfa.final_record || state.is_start() {
if state >= self.dfa.final_record
|| state.is_start()
|| state == self.dfa.in_comment
{
self.dfa.new_state_final_end()
} else {
self.dfa.new_state_final_record()
Expand Down Expand Up @@ -1116,6 +1119,8 @@ struct Dfa {
in_field: DfaState,
/// The DFA state corresponding to being inside an quoted field.
in_quoted: DfaState,
/// The DFA state corresponding to being inside a comment.
in_comment: DfaState,
/// The minimum DFA state that indicates a field has been parsed. All DFA
/// states greater than this are also final-field states.
final_field: DfaState,
Expand All @@ -1132,6 +1137,7 @@ impl Dfa {
classes: DfaClasses::new(),
in_field: DfaState(0),
in_quoted: DfaState(0),
in_comment: DfaState(0),
final_field: DfaState(0),
final_record: DfaState(0),
}
Expand Down Expand Up @@ -1167,6 +1173,7 @@ impl Dfa {
fn finish(&mut self) {
self.in_field = self.new_state(NfaState::InField);
self.in_quoted = self.new_state(NfaState::InQuotedField);
self.in_comment = self.new_state(NfaState::InComment);
self.final_field = self.new_state(NfaState::EndFieldDelim);
self.final_record = self.new_state(NfaState::EndRecord);
}
Expand Down Expand Up @@ -1722,6 +1729,23 @@ mod tests {
b.comment(Some(b'#'));
}
);
// ref: https://github.com/BurntSushi/rust-csv/issues/363
parses_to!(
comment_at_end_of_file_should_be_ignored1,
"foo,bar,baz\n# this is a comment in last line",
csv![["foo", "bar", "baz"]],
|b: &mut ReaderBuilder| {
b.comment(Some(b'#'));
}
);
parses_to!(
comment_at_end_of_file_should_be_ignored2,
"foo,bar,baz\n# this is a comment in last line\n",
csv![["foo", "bar", "baz"]],
|b: &mut ReaderBuilder| {
b.comment(Some(b'#'));
}
);

macro_rules! assert_read {
(
Expand Down