[Feature request] Skip lines until certain text appears

[LupinIII]

Feature Proposal

It would be nice to have an option that skips lines until a certain text appears (in the simplest way at the beginning of the line, or more versatile with a regex). The fixed number for skip lines is too rigid for an application I have. The numeric skipLines option should still be there, after which the, let's call it, skipUntil option checks the lines for a text match. A match would be the first line of the import.

Feature Use Case

I have a CSV file that comes from a banking application. That file contains a few lines with miscellaneous information at the beginning. The problem is that the number of lines is different depending on export settings. The actual CSV-data start at line 7, 8 or 9 with the column headers. The column headers are always the same, which I could match for to mark the start of the data.

[paulrudy]

I just ran into this problem as well. While a feature would be helpful, what I did was to call a separate function to count the number of lines before the actual data is encountered.

import { createReadStream } from "node:fs";
import { createInterface } from "node:readline/promises";
import stripBomStream from 'strip-bom-stream';

export const countEmptyLeadingLines = async ({ inputPath }) => {
	const stream = createReadStream(inputPath).pipe(stripBomStream());

	const rl = createInterface({
		input: stream,
		crlfDelay: Infinity, // Treats \r\n as \n
	});

	let emptyLeadingLines = 0;

	try {
		for await (const line of rl) {
			if (/^(,+|\s*)$/.test(line)) {
				emptyLeadingLines++;
			} else {
				break;
			}
		}
	} catch (error) {
		console.log(error);
	} finally {
		rl.close();
	}

	return emptyLeadingLines;
};

I call this function and use the result to pass to csv-parser's skipLines parameter.

In this case, this script isn't looking for valid data, but instead testing for empty lines or lines with only commas (which is what my bank starts off its exports with). It could be adjusted to instead look for a particular header field. Also, this uses a readable stream, but it's not really necessary unless it's a huge CSV file.