Answer.

Parsing large files and streams "on-the-fly" (one-pass parsing) is an important technique in Perl for log analytics, data processing, packaging, and interacting with external services. One-pass parsers require high efficiency and minimal memory consumption as they do not allow loading the entire file or stream into memory.

Background

Since its inception, Perl has been popular among sysadmins and log analysts due to its powerful string operations and the ability to process gigantic text streams with minimal memory overhead. The use of regular expressions and stream-processing generators has become a standard when building such parsers.

Challenges

The main difficulties include:

avoiding memory leaks
correctly parsing complex patterns on-the-fly
proper error handling
resilience to invalid/partially broken data

Solutions

Basic techniques include:

Using line-by-line reading of files/streams (while (<$fh>) { ... })
For complex parsing logic, gradually accumulating partial results
Parsing lines or blocks only as they arrive

Example code:

open my $fh, '<', 'big.log' or die $!;
while (my $line = <$fh>) {
    next unless $line =~ /^ERROR/;
    if ($line =~ /code=(\d+)/) {
        print "Error code: $1
";
    }
}
close $fh;

Key features:

An array of all lines is not created — data is processed one by one
Flexible ability to skip or terminate processing on partial matches
Composition with files, sockets, pipes, STDIN/STDOUT

Trick questions.

Can slurp (reading the entire file into memory) be safely used when processing one-pass parsers?

No, slurp (reading the entire file into a single string using local $/;) will lead to a sharp increase in memory consumption, which is unacceptable for large files in a high data stream environment.

What is the danger of a simple while (<$fh>) without explicit error handling for reading?

If the reading result is not checked and errors are not handled, you may skip damaged or incomplete lines or lose data during stream failures.

while (defined(my $line = <$fh>)) { ... }

How to properly handle binary and multibyte streams?

Perl works with text files by default. To process binary data, it is important to set binmode for the descriptor: binmode($fh);, and for multibyte UTF-8 streams: binmode($fh, ":encoding(UTF-8)");.

Typical mistakes and anti-patterns

Using slurp when working with large files
Unhandled I/O errors
Violating block boundaries (e.g., when parsing multiline records)

Real-life example

Negative case

A company analyzed logs by reading them entirely through slurp for subsequent splitting into lines. As the volume of data grew, the server began to "die" due to memory shortages on each iteration.

Pros:

Short and understandable code for small files

Cons:

Absolute non-functionality on large logs, increased latency, system crashes

Positive case

An analyst built a chain of one-pass parsers: only interesting events were extracted from each line, and the result was either immediately outputted or aggregated in memory with constraints (e.g., count or sum).