Luaparse

A Lua parser written in JavaScript, for my bachelor's thesis at Arcada.

luaparse Build Status

A Lua parser written in JavaScript, for my bachelor's thesis at Arcada.

Installation

Install through bower install luaparse or npm install luaparse.

Usage

CommonJS

var parser = require('luaparse');
var ast = parser.parse('i = 0');
console.log(JSON.stringify(ast));

AMD

require(['luaparse'], function(parser) {
  var ast = parser.parse('i = 0');
  console.log(JSON.stringify(ast));
});

Browser

<script src="luaparse.js"></script>
<script>
var ast = luaparse.parse('i = 0');
console.log(JSON.stringify(ast));
</script>

Parser Interface

Basic usage:

luaparse.parse(code, options);

The output of the parser is an Abstract Syntax Tree (AST) formatted in JSON.

The available options are:

The default options are also exposed through luaparse.defaultOptions where they can be overriden globally.

There is a second interface which might be preferable when using the wait option.

var parser = luaparse.parse({ wait: true });
parser.write('foo = "');
parser.write('bar');
var ast = parser.end('"');

This would be identical to:

var ast = luaparse.parse('foo = "bar"');

AST format

If the following code is executed:

luaparse.parse('foo = "bar"');

then the returned value will be:

{
  "type": "Chunk",
  "body": [
    {
      "type": "AssignmentStatement",
      "variables": [
        {
          "type": "Identifier",
          "name": "foo"
        }
      ],
      "init": [
        {
          "type": "StringLiteral",
          "value": "bar",
          "raw": "\"bar\""
        }
      ]
    }
  ],
  "comments": []
}

Note on character encodings

Unlike strings in JavaScript, Lua strings are not Unicode strings, but bytestrings (sequences of 8-bit values); likewise, implementations of Lua parse the source code as a sequence of octets. However, the input to this parser is a JavaScript string, i.e. a sequence of 16-bit code units (not necessarily well-formed UTF-16). This poses a problem of how those code units should be interpreted, particularly if they are outside the Basic Latin block ('ASCII').

Currently, this parser handles Unicode input by encoding it in WTF-8, and reinterpreting the resulting code units as Unicode code points. This applies to string literals and (if extendedIdentifiers is enabled) to identifiers as well. Lua byte escapes inside string literals are interpreted directly as code points, while Lua 5.3 \u{} escapes are similarly decoded as UTF-8 code units reinterpreted as code points. It is as if the parser input was being interpreted as ISO-8859-1, while actually being encoded in UTF-8.

This ensures that no otherwise-valid input will be rejected due to encoding errors. Assuming the input was originally encoded in UTF-8 (which includes the case of only containing ASCII characters), it also preserves the following properties:

Maintaining those properties makes the logic of static analysers and code transformation tools simpler. However, it poses a problem when displaying strings to the user and serialising AST back into a string; to recover the original bytestrings, values transformed in this way will have to be encoded in ISO-8859-1.

Other solutions to this problem may be considered in the future. Some of them have been listed below, with their drawbacks:

Custom AST

The default AST structure is somewhat inspired by the Mozilla Parser API but can easily be overriden to customize the structure or to inject custom logic.

luaparse.ast is an object containing all functions used to create the AST, if you for example wanted to trigger an event on node creations you could use the following:

var luaparse = require('luaparse'),
    events = new (require('events').EventEmitter);

Object.keys(luaparse.ast).forEach(function(type) {
  var original = luaparse.ast[type];
  luaparse.ast[type] = function() {
    var node = original.apply(null, arguments);
    events.emit(node.type, node);
    return node;
  };
});
events.on('Identifier', function(node) { console.log(node); });
luaparse.parse('i = "foo"');

this is only an example to illustrate what is possible and this particular example might not suit your needs as the end location of the node has not been determined yet. If you desire events you should use the onCreateNode callback instead).

Lexer

The lexer used by luaparse can be used independently of the recursive descent parser. The lex function is exposed as luaparse.lex() and it will return the next token up until EOF is reached.

Each token consists of:

var parser = luaparse.parse('foo = "bar"', { wait: true });
parser.lex(); // { type: 8, value: "foo", line: 1, lineStart: 0, range: [0, 3] }
parser.lex(); // { type: 32, value: "=", line: 1, lineStart: 0, range: [4, 5]}
parser.lex(); // { type: 2, value: "bar", line: 1, lineStart: 0, range: [6, 11] }
parser.lex(); // { type: 1, value: "<eof>", line: 1, lineStart: 0, range: [11 11] }
parser.lex(); // { type: 1, value: "<eof>", line: 1, lineStart: 0, range: [11 11] }

Examples

Have a look in the examples directory of the repository for some code examples or check them out live.

luaparse(1)

The luaparse executable can be used in your shell by installing luaparse globally using npm:

$ npm install -g luaparse
$ luaparse --help

Usage: luaparse [option]... [file|code]...

Options:
  -c|--code [code]   parse code snippet
  -f|--file [file]   parse from file
  -b|--beautify      output an indenteted AST
  --[no]-comments    store comments. defaults to true
  --[no]-scope       store variable scope. defaults to false
  --[no]-locations   store location data on syntax nodes. defaults to false
  --[no]-ranges      store start and end character locations. defaults to false
  -q|--quiet         suppress output
  -h|--help
  -v|--version
  --verbose

Examples:
  luaparse --no-comments -c "locale foo = \"bar\""
  luaparse foo.lua bar.lua

Example usage

$ luaparse "i = 0"

{"type":"Chunk","body":[{"type":"AssignmentStatement","variables":[{"type":"Identifier","name":"i"}],"init":[{"type":"NumericLiteral","value":0,"raw":"0"}]}],"comments":[]}

Support

Has been tested in at least IE6+, Firefox 3+, Safari 4+, Chrome 10+, Opera 10+, Node 0.4.0+, RingoJS 0.8-0.9, Narwhal 0.3.2, Rhino 1.7R4-1.7R5, Nashorn 1.8.0.

Quality Assurance

TL;DR simply run make qa. This will run all quality assurance scripts but assumes you have it set up correctly.

Begin by cloning the repository and installing the development dependencies with npm install. To test AMD loading for browsers you should run bower install which will download RequireJS.

The luaparse test suite uses testem as a test runner, and because of this it's very easy to run the tests using different javascript engines or even on locally installed browsers. Currently the default runner uses PhantomJS and node so when using make test or npm test you should have PhantomJS installed.

Test runners

Other quality assurance measures

Documentation

By running make docs all documentation will be generated.

Projects using/extending luaparse

Acknowledgements

License

MIT