Ten seconds to one millisecond

DuckDB v2.0 replaces the SQL parser it inherited from PostgreSQL with one built on a parsing expression grammar. Daniël ten Wolde set out the reasoning on 20 August. The new parser covers the full DuckSQL dialect and existing queries continue to parse.

The reason is maintenance, not speed

A YACC/Bison grammar answers grammar changes with shift/reduce and reduce/reduce conflicts, and those conflicts appear in rules the author did not touch. As the post puts it, seemingly small additions can interact with existing rules and introduce conflicts. A PEG grammar tries alternatives in order, so the class of conflict does not arise. What that buys is the ability to keep adding syntax without every addition becoming an archaeology exercise.

Where the 10.6 seconds came from

Ordered alternatives cost backtracking, and backtracking on malformed input goes exponential. Nineteen unmatched opening parentheses took 10.640 seconds to reject. Packrat parsing — caching each matcher's result at each token position — brought the same input to 0.001 seconds.

It is worth being precise about what improved. This is not a query getting faster; it is a syntax error being reported in reasonable time instead of hanging the connection. For anyone running a SQL endpoint that accepts input from a generator or a user, that is the difference between a rejected statement and an availability incident.

What the new grammar made possible

• Expression statements that do not require SELECT.
• CONNECT and DISCONNECT for remote databases.
• CREATE EXTERNAL RESOURCE.
• COPY TO with PARTITION BY and ORDER BY.

The more consequential change is for extensions. They can now inject grammar rules directly instead of shipping a fallback parser of their own, reusing DuckDB's existing transformation machinery — the pipe syntax example in the post works this way. That moves custom SQL surfaces from a maintained fork of the parser to an addition alongside it.