Code

Exhaustive MySQL Parser #157

Code

Merged

adamziel merged 169 commits intodevelopWordPress/sqlite-database-integration:developfrom recursive-descent-parserWordPress/sqlite-database-integration:recursive-descent-parserCopy head branch name to clipboardNov 18, 2024

+126,608-1

Commits

Commits on Aug 17, 2024

• MySQL AST Parser

  adamziel

  committed
  ccc341b

  Copy full SHA for ccc341b

• Fix parser overriding parts of the parse tree as it constructs them.

  adamziel

  committed
  78fdf69

  Copy full SHA for 78fdf69

• Output ParseTree using a class, not an array for much simpler processing

  Show description for c8652d5
  adamziel

  committed
  c8652d5

  Copy full SHA for c8652d5

Commits on Aug 18, 2024

• Manually factor left recursion into right recursion in the grammar file for a more useful parse tree, 15x smaller grammar file, and faster parsing time

  adamziel

  committed
  137d6ca

  Copy full SHA for 137d6ca

Commits on Aug 20, 2024

• Explore support for SQL_CALC_FOUND_ROWS

  adamziel

  committed
  0a2440c

  Copy full SHA for 0a2440c

• Support VALUES() call

  adamziel

  committed
  0406d71

  Copy full SHA for 0406d71

Commits on Sep 26, 2024

• Extract queries from MySQL test suite and test the parser against them

  JanJakes

  committed
  87573f2

  Copy full SHA for 87573f2

• Implement handling for manually added lexer symbols

  JanJakes

  committed
  9629702

  Copy full SHA for 9629702

• Fix passing nulls to "ctype_" functions

  JanJakes

  committed
  d63bc6e

  Copy full SHA for d63bc6e

• Add support for hex format x'ab12', X'ab12', and bin format x'01' and X'01'

  JanJakes

  committed
  8e7e2e8

  Copy full SHA for 8e7e2e8

• Fix wrong MySQL version conditions (AI hallucinations)

  JanJakes

  committed
  ebcc17e

  Copy full SHA for ebcc17e

• Implement the checkCharset() placeholder function

  JanJakes

  committed
  1551b0e

  Copy full SHA for 1551b0e

• Document manual grammar factoring

  JanJakes

  committed
  cdd84b4

  Copy full SHA for cdd84b4

• Fix "alterOrderList" that has a wrong definition in the original grammar

  JanJakes

  committed
  f50b515

  Copy full SHA for f50b515

• Fix "createUser" that was incorrectly converted from ANTLR to EBNF

  JanJakes

  committed
  e267f67

  Copy full SHA for e267f67

• Fix "castType" that was incomplete in the original grammar

  JanJakes

  committed
  cd543af

  Copy full SHA for cd543af

• Fix "SELECT ... WHERE ... INTO @var" using a negative lookahead

  JanJakes

  committed
  135f29f

  Copy full SHA for 135f29f

Commits on Sep 27, 2024

• Fix "EXPLAIN FORMAT=..." by reordering grammar rules

  JanJakes

  committed
  27524dd

  Copy full SHA for 27524dd

• Fix special "WINDOW" and "OVER" cases by adjusting grammar rules

  JanJakes

  committed
  069342f

  Copy full SHA for 069342f

• Fix "GRANT" and "REVOKE" by adjusting grammar rules to solve conflicts

  JanJakes

  committed
  9bfc977

  Copy full SHA for 9bfc977

• Use ebnfutils to dump grammar conflicts

  JanJakes

  committed
  ca4de77

  Copy full SHA for ca4de77

Commits on Sep 30, 2024

• Implement the determineFunction() placeholder function, unify SQL modes

  JanJakes

  committed
  cd3504d

  Copy full SHA for cd3504d

• Fix processing NOW() synonyms in lexer

  Show description for 81bbde0
  JanJakes

  committed
  81bbde0

  Copy full SHA for 81bbde0

• Match mysqltest commands case-insensitively

  JanJakes

  committed
  71292fb

  Copy full SHA for 71292fb

Commits on Oct 1, 2024

• Add a script to test lexer on all the testing queries

  JanJakes

  committed
  1ab3723

  Copy full SHA for 1ab3723

Commits on Oct 2, 2024

• Replace lexer switch/case and function calls with lookup tables

  Show description for 42ffc1b
  JanJakes

  committed
  42ffc1b

  Copy full SHA for 42ffc1b

• Fix unicode handling when extracting test queries

  JanJakes

  committed
  01241b8

  Copy full SHA for 01241b8

• Fix identifier matching, improve lexer performance by ~25%

  JanJakes

  committed
  f5266f1

  Copy full SHA for f5266f1

• Unify charset matching with identifier matching, remove non-existent symbol

  JanJakes

  committed
  a2ac60b

  Copy full SHA for a2ac60b

• Fix quoted text/identifier matching, improve lexer performance by ~10%

  JanJakes

  committed
  90e2af6

  Copy full SHA for 90e2af6

Commits on Oct 3, 2024

• Remove dependency on ctype, improve lexer performance by ~4%

  JanJakes

  committed
  dfcad3e

  Copy full SHA for dfcad3e

• Determine token names lazily, reduce LOC to < 3000, improve lexer performance by ~4%

  Show description for 0f23dd3
  JanJakes

  committed
  0f23dd3

  Copy full SHA for 0f23dd3

• Finish manual token pass, use MySQL Workbench token IDs, add comments

  JanJakes

  committed
  6bb8ff9

  Copy full SHA for 6bb8ff9

• Fix wrong token type

  JanJakes

  committed
  ec55c10

  Copy full SHA for ec55c10

• Inline some simple single-use methods, remove unused methods

  JanJakes

  committed
  21f3f16

  Copy full SHA for 21f3f16

• Move token iteration loop outside nextToken() method

  JanJakes

  committed
  b4f0e08

  Copy full SHA for b4f0e08

• Fix and simplify lookahead logic, improve lexer performance by ~6%

  JanJakes

  committed
  f658751

  Copy full SHA for f658751

• Fix date and time literals by reordering grammar rules

  JanJakes

  committed
  4eee991

  Copy full SHA for 4eee991

Commits on Oct 4, 2024

• Implement handling of unquoted user variable suffixes

  JanJakes

  committed
  64e2068

  Copy full SHA for 64e2068

• Fix number vs identifier matching, add some tests

  JanJakes

  committed
  f1c56cc

  Copy full SHA for f1c56cc

• Fix handling of identifiers preceded by a “.”

  JanJakes

  committed
  883c08a

  Copy full SHA for 883c08a

• Implement lexeer support for NCHAR literals

  JanJakes

  committed
  f56f036

  Copy full SHA for f56f036

• Fix administration statements to support both "TABLE" and "TABLES" keywords

  JanJakes

  committed
  972f7e0

  Copy full SHA for 972f7e0

• Use MySQL 8.0.38 test suite, the same version as the grammar

  JanJakes

  committed
  df65873

  Copy full SHA for df65873

Commits on Oct 5, 2024

• Use the grammar MySQL version 8.0.38 as a default also in the lexer

  JanJakes

  committed
  f25b4fa

  Copy full SHA for f25b4fa

• Add a missing charset from MySQL 5, improve comments

  JanJakes

  committed
  1b53df5

  Copy full SHA for 1b53df5

• Sort version-specific keyword list alphabetically

  JanJakes

  committed
  905a7d1

  Copy full SHA for 905a7d1

• Fix conflict between “<index> USING” and “<index> TYPE” by adjusting grammar rules

  JanJakes

  committed
  7d1cdd7

  Copy full SHA for 7d1cdd7

Commits on Oct 15, 2024

• Fix “LIKE …” and “LIKE (…)” by reordering grammar rules

  JanJakes

  committed
  4e5e499

  Copy full SHA for 4e5e499

• Fix "sumExpr" (AVG, SUM, COUNT, …) by reordering grammar rules

  JanJakes

  committed
  4eb9dd5

  Copy full SHA for 4eb9dd5

• Fix "REVOKE ALL ON ... FROM ..." by adding a missing rule segment

  JanJakes

  committed
  4f968cd

  Copy full SHA for 4f968cd

Commits on Oct 16, 2024

• Add full MySQLParser.g4 grammar

  Show description for 6188d90
  JanJakes

  committed
  6188d90

  Copy full SHA for 6188d90

• Add manual fixes to MySQLParser.g4 grammar

  Show description for b371daf
  JanJakes

  committed
  b371daf

  Copy full SHA for b371daf

• Add a script to parse grammar directly from MySQLParser.g4

  Show description for 1765e4f
  JanJakes

  committed
  1765e4f

  Copy full SHA for 1765e4f

• Handle empty branches during grammr parsing

  JanJakes

  committed
  50ef591

  Copy full SHA for 50ef591

• Place and document grammar fixes below the original rules

  JanJakes

  committed
  28e44f5

  Copy full SHA for 28e44f5

• Skip perl, append_file, and write_file commands in query extraction

  JanJakes

  committed
  cd3c335

  Copy full SHA for cd3c335

• Fix “histogram” rule by adding missing “USING DATA” clause

  JanJakes

  committed
  09bd0c7

  Copy full SHA for 09bd0c7

Commits on Oct 17, 2024

• Fix conflict in "explainStatement" by reordering grammar rules

  JanJakes

  committed
  d06be19

  Copy full SHA for d06be19

• Handle escaped quotes when parsing test queries

  JanJakes

  committed
  cebfdd7

  Copy full SHA for cebfdd7

• Fix "alterCommandList" to solve conflicts between "alterCommandsModifierList" prefixes

  JanJakes

  committed
  94b7bd2

  Copy full SHA for 94b7bd2

• Add missing column visibility settings to the grammar

  JanJakes

  committed
  ba43fc7

  Copy full SHA for ba43fc7

• Fix "fieldDefinition" to solve conflict between "columnAttribute" and GENERATED/AS

  JanJakes

  committed
  03fbb98

  Copy full SHA for 03fbb98

• Fix "replicationStatement" to correctly support the "RESET PERSIST" statement

  JanJakes

  committed
  ed15ffa

  Copy full SHA for ed15ffa

• Implement missing "EXCEPT" and "INTERSECT" operators in the grammar

  JanJakes

  committed
  0b16f44

  Copy full SHA for 0b16f44

Commits on Oct 18, 2024

• Skip if and while blocks when parsing tests, skip mysqltest.test

  JanJakes

  committed
  8df7294

  Copy full SHA for 8df7294

• Fix processing “--“ commands and “--delimiter”

  JanJakes

  committed
  0a35989

  Copy full SHA for 0a35989

• Fix "alterTableActions" to solve conflicts between "alterCommandsModifierList" and "alterCommandList"

  JanJakes

  committed
  442a682

  Copy full SHA for 442a682

• Fix REVOKE statement grammar

  JanJakes

  committed
  207078a

  Copy full SHA for 207078a

• Add missing branches to “accountLockPasswordExpireOptions”

  JanJakes

  committed
  56c7b8d

  Copy full SHA for 56c7b8d

• Fix ALTER USER statement grammar

  JanJakes

  committed
  9bdd088

  Copy full SHA for 9bdd088

• Fix "userIdentifierOrText" to support omitting sequence after "@"

  JanJakes

  committed
  8f54a3f

  Copy full SHA for 8f54a3f

Commits on Oct 25, 2024

• Fix conflict within "queryExpressionParens"

  JanJakes

  committed
  724e1c3

  Copy full SHA for 724e1c3

• Fix CHARACTER VARYING support by reordering “dataType” subrules

  JanJakes

  committed
  6e985b2

  Copy full SHA for 6e985b2

• Fix ALL/ANY support by reordering subrules

  JanJakes

  committed
  e2e7501

  Copy full SHA for e2e7501

Commits on Oct 28, 2024

• Implement MySQL-specific and version comments: /*!… */

  Show description for 61d2bcc
  JanJakes

  committed
  61d2bcc

  Copy full SHA for 61d2bcc

• Fix SHOW GRANTS … USING by reordering grammar rules

  JanJakes

  committed
  3118f11

  Copy full SHA for 3118f11

• Fix query expressions followed by “INTO …”

  JanJakes

  committed
  cb44ea7

  Copy full SHA for cb44ea7

• Fix “SELECT … FOR UPDATE … INTO …” by adding a missing rule part

  JanJakes

  committed
  0812288

  Copy full SHA for 0812288

• Fix “DO … AS …” by reordering rules

  JanJakes

  committed
  d5e1abb

  Copy full SHA for d5e1abb

• Fix PARTITION being defined as a non-reserved keyword

  Show description for e3555d2
  JanJakes

  committed
  e3555d2

  Copy full SHA for e3555d2

• Fix handling —error in MySQL test files

  JanJakes

  committed
  abdbe47

  Copy full SHA for abdbe47

• Fix ignoring MySQL test commands without params

  JanJakes

  committed
  5faef36

  Copy full SHA for 5faef36

• Add missing MySQL test commands

  JanJakes

  committed
  53cb169

  Copy full SHA for 53cb169

• Fix handling prefixed quotes in MySQL tests

  JanJakes

  committed
  ea6d067

  Copy full SHA for ea6d067

Commits on Oct 29, 2024

• Add support for missing "INSTALL COMPONENT" statement "SET ..." suffix

  JanJakes

  committed
  22ac89b

  Copy full SHA for 22ac89b

• Fix "indexHintList" to use only whitespace as a separator (not commas)

  JanJakes

  committed
  ce35d1d

  Copy full SHA for ce35d1d

• Fix and simplify line comment handling in lexer

  JanJakes

  committed
  a33723d

  Copy full SHA for a33723d

• Add support for "CHANGE REPLICATION SOURCE ..." statement

  JanJakes

  committed
  eb1e756

  Copy full SHA for eb1e756

• Mark ATTRIBUTE keyword as non-reserved

  JanJakes

  committed
  96accd0

  Copy full SHA for 96accd0

• Fix START TRANSACTION statement with multiple characteristics

  JanJakes

  committed
  f0c97bd

  Copy full SHA for f0c97bd

• Fix handling column attributes by reordering rules

  JanJakes

  committed
  d861ba7

  Copy full SHA for d861ba7

• Fix GRANT “role with @“ by reordering grammar rules

  JanJakes

  committed
  c74dbca

  Copy full SHA for c74dbca

• Fix GRANT CREATE/DROP ROLE by reordering rules

  JanJakes

  committed
  143c421

  Copy full SHA for 143c421

• Add support for COMMENT and ATTRIBUTE to CREATE USER

  JanJakes

  committed
  a48810c

  Copy full SHA for a48810c

• Fix FLUSH TABLES with qualified identifiers

  JanJakes

  committed
  a80dbf1

  Copy full SHA for a80dbf1

• Fix temporal literals to support double quotes

  JanJakes

  committed
  f6ebd7a

  Copy full SHA for f6ebd7a

• Add support for CAST(... AT TIME ZONE ... AS DATETIME ...)

  JanJakes

  committed
  09cdac7

  Copy full SHA for 09cdac7

• Fix transaction characteristics separator

  JanJakes

  committed
  05d02ec

  Copy full SHA for 05d02ec

• Add support for IF NOT EXISTS for function, procedure, and trigger

  JanJakes

  committed
  8022fc5

  Copy full SHA for 8022fc5

• Fix SET PASSWORD statement conflicts

  JanJakes

  committed
  e98caad

  Copy full SHA for e98caad

• Fix handling nchar, national character, etc.

  JanJakes

  committed
  47b7ec4

  Copy full SHA for 47b7ec4

• Ignore “query_attributes” MySQL client command

  JanJakes

  committed
  522867d

  Copy full SHA for 522867d

Commits on Oct 30, 2024

• Fix "leadLagInfo" to support identifiers and variables as well

  JanJakes

  committed
  0d80a16

  Copy full SHA for 0d80a16

• Fix WEIGHT_STRING(… AS BINARY(…)) by reordering grammar rules

  JanJakes

  committed
  d12d01e

  Copy full SHA for d12d01e

• Add support for "ALTER INSTANCE ..." statement

  JanJakes

  committed
  ad62b8a

  Copy full SHA for ad62b8a

• Fix CAST(… AS FLOAT(N)) by fixing a wrong grammar rule

  JanJakes

  committed
  560064a

  Copy full SHA for 560064a

• Add missing CREATE TABLE options

  JanJakes

  committed
  d004dad

  Copy full SHA for d004dad

• Add support for ON DELETE SET DEFAULT

  JanJakes

  committed
  52d9ed1

  Copy full SHA for 52d9ed1

• Add support for ENGINE_ATTRIBUTE to tablespaces

  JanJakes

  committed
  fb1e5cf

  Copy full SHA for fb1e5cf

• Add support for "JSON_VALUE(..., '...' RETURNING <type>)

  JanJakes

  committed
  e5f59fe

  Copy full SHA for e5f59fe

• Skip disabled MySQL tests

  JanJakes

  committed
  106a6e5

  Copy full SHA for 106a6e5

• Detect & convert charset when extracting tests

  JanJakes

  committed
  e5d3a03

  Copy full SHA for e5d3a03

• Fix lexing numbers and identifiers

  JanJakes

  committed
  88e3ec2

  Copy full SHA for 88e3ec2

• Fix CREATE/ALTER TABLE … UNION = ()

  JanJakes

  committed
  c934f37

  Copy full SHA for c934f37

• Improve RESET PERSIST fix

  JanJakes

  committed
  dfee145

  Copy full SHA for dfee145

• Fix ALTER DABATASE to support optional schema name

  JanJakes

  committed
  1c7c18c

  Copy full SHA for 1c7c18c

Commits on Oct 31, 2024

• Fix support for “:=“ assignment in “updateElement”

  JanJakes

  committed
  ae0b31a

  Copy full SHA for ae0b31a

• Add “phpize-grammar” logic to grammar conversion script

  JanJakes

  committed
  839d807

  Copy full SHA for 839d807

• Delete all no longer necessary grammar tools

  JanJakes

  committed
  495e227

  Copy full SHA for 495e227

• Use WP coding styles for all new files

  JanJakes

  committed
  d9827ac

  Copy full SHA for d9827ac

• Ignore coding styles for compressed grammar file

  JanJakes

  committed
  a19d439

  Copy full SHA for a19d439

• Use WP coding styles for filenames, use file per class, improve directory structure

  JanJakes

  committed
  8d810fe

  Copy full SHA for 8d810fe

• Exclude new parser tests for now

  JanJakes

  committed
  d869785

  Copy full SHA for d869785

• Remove no longer needed rule name processing

  Show description for f3f0c0f
  JanJakes

  committed
  f3f0c0f

  Copy full SHA for f3f0c0f

• Omit grammar tools and WIP work from plugin builds

  JanJakes

  committed
  d993041

  Copy full SHA for d993041

• Omit all test files from plugin builds

  JanJakes

  committed
  8c2d227

  Copy full SHA for 8c2d227

• Omit new lexer & parser from plugin builds for now

  JanJakes

  committed
  2cf39d4

  Copy full SHA for 2cf39d4

Commits on Nov 5, 2024

• Improve class naming and directory structure

  JanJakes

  committed
  09dda8a

  Copy full SHA for 09dda8a

• Move tokenize_query to WP_MySQL_Lexer::tokenize()

  JanJakes

  committed
  e42bf52

  Copy full SHA for e42bf52

• Move tests under mysql directory

  JanJakes

  committed
  c42ee15

  Copy full SHA for c42ee15

Commits on Nov 6, 2024

• Move manual lexer tests to PHPUnit

  JanJakes

  committed
  7b84cfc

  Copy full SHA for 7b84cfc

• Move test tools to tests/tools and ignore them in PHPUnit

  JanJakes

  committed
  be7257f

  Copy full SHA for be7257f

• Polish and document test downloading and extraction scripts

  JanJakes

  committed
  9c34178

  Copy full SHA for 9c34178

• Skip queries testing parser stack overflow

  JanJakes

  committed
  c2cfd45

  Copy full SHA for c2cfd45

• Run the full MySQL server test suite in unit tests

  JanJakes

  committed
  03cc66a

  Copy full SHA for 03cc66a

• Repurpose testing scripts for benchmarking, add single-parse testing script

  JanJakes

  committed
  5b2f488

  Copy full SHA for 5b2f488

• Use strcspn to handle all types of quoted text

  JanJakes

  committed
  1e3e254

  Copy full SHA for 1e3e254

• Use strspn to handle unquoted user-defined variables

  JanJakes

  committed
  7d5869b

  Copy full SHA for 7d5869b

Commits on Nov 7, 2024

• Replace preg_match with strspn and UTF-8 decoder when handling identifiers

  JanJakes

  committed
  04bf6e5

  Copy full SHA for 04bf6e5

Commits on Nov 8, 2024

• Simplify lexer state, use and advance only byte position

  JanJakes

  committed
  43357d0

  Copy full SHA for 43357d0

• Replace all other byte scanning loops with strspn

  JanJakes

  committed
  6e3b31e

  Copy full SHA for 6e3b31e

• Inline bin and hex numbers to number method

  JanJakes

  committed
  3b789fe

  Copy full SHA for 3b789fe

• Simplify version comment processing

  JanJakes

  committed
  ad4cf36

  Copy full SHA for ad4cf36

• Simplify matching EOF

  JanJakes

  committed
  3730d80

  Copy full SHA for 3730d80

• Remove channel, token, and token instance properties

  JanJakes

  committed
  833ca68

  Copy full SHA for 833ca68

Commits on Nov 11, 2024

• Replace type property with local variables and function returns

  JanJakes

  committed
  e0dfccd

  Copy full SHA for e0dfccd

• Improve property and method naming and documentation

  JanJakes

  committed
  dff4649

  Copy full SHA for dff4649

• Check for the U+0080-U+FFFF range manually, add test coverage

  JanJakes

  committed
  17641f7

  Copy full SHA for 17641f7

Commits on Nov 12, 2024

• Improve lexer docs and naming

  JanJakes

  committed
  ee9d59d

  Copy full SHA for ee9d59d

• Unify identifier matching with other "read_" methods

  JanJakes

  committed
  b32a9dd

  Copy full SHA for b32a9dd

• Reorder methods to group "read_" methods together

  JanJakes

  committed
  432bf1f

  Copy full SHA for 432bf1f

• Implement integer type detection

  JanJakes

  committed
  73c6c0a

  Copy full SHA for 73c6c0a

• Move empty rule (ε) constant to WP_Parser_Grammar

  JanJakes

  committed
  4464cea

  Copy full SHA for 4464cea

• Cleanup, add docs and TODOs

  JanJakes

  committed
  79b2622

  Copy full SHA for 79b2622

• Rename WP_Parser_Tree to WP_Parser_Node and document it

  JanJakes

  committed
  43cec5d

  Copy full SHA for 43cec5d

• Explain the "ε" rule in a comment

  JanJakes

  committed
  634635e

  Copy full SHA for 634635e

• Rename $rule to $branches for better clarity

  JanJakes

  committed
  96dd467

  Copy full SHA for 96dd467

• Use false rather than null when a parser subtree doesn't match

  JanJakes

  committed
  dec0e3f

  Copy full SHA for dec0e3f

• Declare mbstring in dev dependencies

  JanJakes

  committed
  afc70bb

  Copy full SHA for afc70bb

Commits on Nov 13, 2024

• Use more descriptive file name for MySQL test queries

  JanJakes

  committed
  33d55c4

  Copy full SHA for 33d55c4

• Read quote from the SQL payload instead of passing it as parameter

  JanJakes

  committed
  dd31359

  Copy full SHA for dd31359

• Improve code comments

  JanJakes

  committed
  ae42217

  Copy full SHA for ae42217

• Simplify next_token() loop condition

  JanJakes

  committed
  5942fd6

  Copy full SHA for 5942fd6

• Inline is-digit and is-whitespace logic

  JanJakes

  committed
  c567a3f

  Copy full SHA for c567a3f

• Fix 0b and 0x indentifiers

  JanJakes

  committed
  60d995d

  Copy full SHA for 60d995d

• Fix unclosed b'...' and x'...' numbers

  JanJakes

  committed
  ccebf37

  Copy full SHA for ccebf37

• Rename parser testing script to dump-ast.php

  JanJakes

  committed
  7155b7b

  Copy full SHA for 7155b7b

Commits on Nov 14, 2024

• Implement "next_token()" & "get_next_token" API

  JanJakes

  committed
  9426969

  Copy full SHA for 9426969