Use library for vimscript parsing instead of regexes

[dbarnett]

Per neovim/neovim#170 and https://neovim.io/doc/user/api.html#nvim_parse_expression(), libnvim now offers an API to parse vimscript syntax to an AST. We should give it a shot hooking that up to vimdoc and see if it helps clean up some of our current parsing quirks.

[dbarnett]

I did a little poking around here and it doesn't look too promising so far. It seems to parse expressions only, not commands, and doesn't seem to handle comments or line continuations either.

Here are some samples of output I got invoking it from inside vim

:echo nvim_parse_expression("s:plugin.Flag('foo', 'bar')", '', 0)
{'ast':
 {'children':
  [{'children':
    [{'ident': 'plugin',
      'len': 8,
      'scope': 115,
      'start': [0, 0],
      'type': 'PlainIdentifier'},
     {'ident': 'Flag', 'len': 4, 'start': [0, 9], 'type': 'PlainKey'}],
    'len': 1,
    'start': [0, 8],
    'type': 'ConcatOrSubscript'},
   {'children':
    [{'len': 5,
      'start': [0, 14],
      'svalue': 'foo',
      'type': 'SingleQuotedString'},
     {'len': 6,
      'start': [0, 20],
      'svalue': 'bar',
      'type': 'SingleQuotedString'}],
    'len': 1,
    'start': [0, 19],
    'type': 'Comma'}],
  'len': 1,
  'start': [0, 13],
  'type': 'Call'},
 'len': 27}

:Verbose PP nvim_parse_expression("call s:plugin.Flag('foo', 'bar')", '', 0)
{'ast':
 {'children':
  [{'ident': 'call', 'len': 4, 'scope': 0, 'start': [0, 0], 'type': 'PlainIdentifier'},
   {'children':
    [{'children':
      [{'ident': 'plugin', 'len': 9, 'scope': 115, 'start': [0, 4], 'type': 'PlainIdentifier'},
       {'ident': 'Flag', 'len': 4, 'start': [0, 14], 'type': 'PlainKey'}],
      'len': 1,
      'start': [0, 13],
      'type': 'ConcatOrSubscript'},
     {'children':   
      [{'len': 5, 'start': [0, 19], 'svalue': 'foo', 'type': 'SingleQuotedString'},
       {'len': 6, 'start': [0, 25], 'svalue': 'bar', 'type': 'SingleQuotedString'}],
      'len': 1,
      'start': [0, 24],
      'type': 'Comma'}],
    'len': 1,
    'start': [0, 18],
    'type': 'Call'}],  
  'len': 0,
  'start': [0, 4],
  'type': 'OpMissing'},
 'error': {'arg': "s:plugin.Flag('foo', 'bar')", 'message': 'E15: Missing operator: %.*s'},
 'len': 32}

:Verbose PP nvim_parse_expression("\"\n\" @public\ncall s:plugin.Flag(\n    \\ 'foo', 'bar')", '', 0)
{'ast':
 {'children':
  [{'len': 3, 'start': [0, 0], 'svalue': "\n", 'type': 'DoubleQuotedString'},
   {'children':
    [{'len': 3, 'name': 112, 'start': [0, 3], 'type': 'Register'},
     {'ident': 'ublic', 'len': 5, 'scope': 0, 'start': [0, 6], 'type': 'PlainIdentifier'}],    
    'len': 0,
    'start': [0, 6],
    'type': 'OpMissing'}],
  'len': 0,
  'start': [0, 3],  
  'type': 'OpMissing'},
 'error': {'arg': "@public\ncall s:plugin.Flag(\n    \\ 'foo', 'bar')", 'message': 'E15: Missing operator: %.*s'},
 'len': 11}

[dbarnett]

I did find that https://github.com/vim-jp/vim-vimlparser does a pretty good job parsing arbitrary vimscript. We'd probably want to use that instead, but I can't figure out how to declare a dependency on it for fetch it from pypi yet (vim-jp/vim-vimlparser#170).

[martindemello]

there's a treesitter parser for viml and python bindings to treesitter - might be worth exploring.