summaryrefslogtreecommitdiff
path: root/Libraries/LibJS/Lexer.cpp
AgeCommit message (Collapse)Author
2020-11-25LibJS: Fix possible OOB read during Lexer constructionLinus Groh
The Lexer constructor calls consume() once, which initializes m_position to be > 0 and sets m_character. consume() calls is_line_terminator(), which wasn't accounting for this state.
2020-10-29LibJS: "-->" preceded by token on same line isn't start of HTML-like commentLinus Groh
B.1.3 HTML-like Comments The syntax and semantics of 11.4 is extended as follows except that this extension is not allowed when parsing source code using the goal symbol Module: Syntax (only relevant part included) SingleLineHTMLCloseComment :: LineTerminatorSequence HTMLCloseComment HTMLCloseComment :: WhiteSpaceSequence[opt] SingleLineDelimitedCommentSequence[opt] --> SingleLineCommentChars[opt] Fixes #3810.
2020-10-26LibJS: Emit token message for invalid numeric literalsLinus Groh
2020-10-26LibJS: Emit TokenType::Invalid for unterminated multi-line commentsLinus Groh
2020-10-26LibJS: Add message string to TokenLinus Groh
This allows us to communicate details about invalid tokens to the parser without having to invent a bunch of specific invalid tokens like TokenType::InvalidNumericLiteral.
2020-10-22LibJS: Support all line terminators (LF, CR, LS, PS)Linus Groh
https://tc39.es/ecma262/#sec-line-terminators
2020-10-19LibJS: Unprefixed octal numbers are a syntax error in strict modeLinus Groh
2020-10-18LibJS: Fix parsing of invalid numeric literalsStephan Unverwerth
i.e. "1e" "0x" "0b" "0o" used to be parsed as valid literals. They now produce invalid tokens. Fixes #3716
2020-10-17LibJS: Avoid creating temporary Strings to look up tokens while lexingAndreas Kling
It would be cool to solve this in a general way so that looking up a string literal or StringView in a HashMap with String keys avoids creating a temp string. For now, this patch simply addresses the issue in JS::Lexer. This is a 2-3% speed-up on test-js.
2020-10-05LibJS: Implement logical assignment operators (&&=, ||=, ??=)Linus Groh
TC39 proposal, stage 4 as of 2020-07. https://tc39.es/proposal-logical-assignment/
2020-09-12LibJS: Fix start position of multi-line tokensBen Wiederhake
This broke in case of unterminated regular expressions, causing goofy location numbers, and 'source_location_hint' to eat up all memory: Unexpected token UnterminatedRegexLiteral. Expected statement (line: 2, column: 4294967292)
2020-06-08LibJS: Move regex logic to main Lexer if statementMatthew Olsson
This prevents a regex such as /=/ from lexing into TokenType::SlashEquals, preventing the regex logic from working.
2020-06-08LibJS: Properly consume escaped backslash in regex literalMatthew Olsson
2020-06-07LibJS: Fix big int division lexing as UnterminatedRegexLiteralMatthew Olsson
2020-06-07LibJS: Add BigIntLinus Groh
2020-06-07LibJS: Lex and parse regex literals, add RegExp objectsMatthew Olsson
This adds regex parsing/lexing, as well as a relatively empty RegExpObject. The purpose of this patch is to allow the engine to not get hung up on parsing regexes. This will aid in finding new syntax errors (say, from google or twitter) without having to replace all of their regexes first!
2020-05-26LibJS: Fix incorrect token column values (#2401)Paul Redmond
- initializing m_line_column to 1 in the lexer results in incorrect column values in tokens on the first line of input. - not incrementing m_line_column when EOF is reached results in an incorrect column value on the last token.
2020-05-15LibJS: Remove syntax errors from lexerLinus Groh
Giving the lexer the ability to generate errors adds unnecessary complexity - also it only calls its syntax_error() function in one place anyway ("unterminated string literal"). But since the lexer *also* emits tokens like Eof or UnterminatedStringLiteral, it should be up to the consumer of these tokens to decide what to do. Also remove the option to not print errors to stderr as that's not relevant anymore.
2020-05-12LibJS: Add missing keywords/tokensLinus Groh
Some of these are required for syntax we have not implemented yet, some are future reserved words in strict mode.
2020-05-05LibJS: Implement exponentiation assignment operator (**=)Linus Groh
2020-05-05LibJS: Implement bitwise assignment operators (&=, |=, ^=)Linus Groh
2020-05-04LibJS: Add template literalsmattco98
Adds fully functioning template literals. Because template literals contain expressions, most of the work has to be done in the Lexer rather than the Parser. And because of the complexity of template literals (expressions, nesting, escapes, etc), the Lexer needs to have some template-related state. When entering a new template literal, a TemplateLiteralStart token is emitted. When inside a literal, all text will be parsed up until a '${' or '`' (or EOF, but that's a syntax error) is seen, and then a TemplateLiteralExprStart token is emitted. At this point, the Lexer proceeds as normal, however it keeps track of the number of opening and closing curly braces it has seen in order to determine the close of the expression. Once it finds a matching curly brace for the '${', a TemplateLiteralExprEnd token is emitted and the state is updated accordingly. When the Lexer is inside of a template literal, but not an expression, and sees a '`', this must be the closing grave: a TemplateLiteralEnd token is emitted. The state required to correctly parse template strings consists of a vector (for nesting) of two pieces of information: whether or not we are in a template expression (as opposed to a template string); and the count of the number of unmatched open curly braces we have seen (only applicable if the Lexer is currently in a template expression). TODO: Add support for template literal newlines in the JS REPL (this will cause a syntax error currently): > `foo > bar` 'foo bar'
2020-05-01LibJS: Implement (no-op) debugger statementLinus Groh
2020-04-27LibJS: Add spreading in array literalsmattco98
Implement the syntax and behavor necessary to support array literals such as [...[1, 2, 3]]. A type error is thrown if the target of the spread operator does not evaluate to an array (though it should eventually just check for an iterable). Note that the spread token's name is TripleDot, since the '...' token is used for two features: spread and rest. Calling it anything involving 'spread' or 'rest' would be a bit confusing.
2020-04-24LibJS: Add TokenType::TemplateLiteralLinus Groh
This is required for template literals - we're not quite there yet, but at least the parser can now tell us when this token is encountered - currently this yields "Unexpected token Invalid". Not really helpful. The character is a "backtick", but as we already have TokenType::{StringLiteral,RegexLiteral} this seemed like a fitting name. This also enables syntax highlighting for template literals in the js REPL and LibGUI's JSSyntaxHighlighter.
2020-04-14LibJS: Handle HTML-style commentsStephan Unverwerth
2020-04-13LibJS: Parse "this" as ThisExpressionStephan Unverwerth
2020-04-05LibJS: Report the start position of a token as its line columnAnotherTest
2020-04-05LibJS: Allow lexer to run without logging errorsAnotherTest
2020-04-05LibJS: Add numeric literal parsing for different bases and exponentsStephan Unverwerth
2020-04-05LibJS: Plumb line and column information through Lexer / ParserBrian Gianforcaro
While debugging test failures, it's pretty frustrating to have to go do printf debugging to figure out what test is failing right now. While watching your JS Raytracer stream it seemed like this was pretty furstrating as well. So I wanted to start working on improving the diagnostics here. In the future I hope we can eventually be able to plumb the info down to the Error classes so any thrown exceptions will contain enough metadata to know where they came from.
2020-04-05LibJS: Add support for "continue" inside "for" statements :^)Andreas Kling
2020-04-04LibJS: Hack the lexer to allow numbers with decimalsAndreas Kling
This is very hackish and should definitely be improved. :^)
2020-04-03LibJS: Remove UndefinedLiteral, add undefined to global objectLinus Groh
There is no such thing as a "undefined literal" in JS - undefined is just a property on the global object with a value of undefined. This is pretty similar to NaN. var undefined = "foo"; is a perfectly fine AssignmentExpression :^)
2020-03-30LibJS: Add support for arrow functionsJack Karamanian
2020-03-29LibJS: Lexer and parser support for "switch" statementsAndreas Kling
2020-03-24LibJS: Implement "throw"Andreas Kling
You can now throw an expression to the nearest catcher! :^) To support throwing arbitrary values, I added an Exception class that sits as a wrapper around whatever is thrown. In the future it will be a logical place to store a call stack.
2020-03-23LibJS: Teach the lexer to recognize ">=" and "<=" :^)Andreas Kling
2020-03-21LibJS: Parse object expressions0xtechnobabble
2020-03-16LibJS: Implement null and undefined literals0xtechnobabble
2020-03-14LibJS: Lex single quote strings, escaped chars and unterminated stringsStephan Unverwerth
2020-03-14LibJS: Add operator precedence parsingStephan Unverwerth
Obey precedence and associativity rules when parsing expressions with chained operators.
2020-03-13LibJS: Fix endless loop in string lexingOriko
2020-03-13LibJS: Fix lexing of the last character in a fileStephan Unverwerth
Before this commit the last character in a file would be swallowed. This also fixes parsing of empty files which would previously ASSERT.
2020-03-12LibJS: Fix some coding style mistakes in LexerAndreas Kling
2020-03-12LibJS: Implement for statementConrad Pankoff
2020-03-12LibJS: Parse === and !== binary operatorsConrad Pankoff
2020-03-12LibJS: Implement basic lexing + parsing of StringLiteralAndreas Kling
This still includes the double-quote characters (") but at least the AST comes out right.
2020-03-12LibJS: Add Javascript lexer and parserStephan Unverwerth
This adds a basic Javascript lexer and parser. It can parse the currently existing demo programs. More work needs to be done to turn it into a complete parser than can parse arbitrary JS Code. The lexer outputs tokens with preceeding whitespace and comments in the trivia member. This should allow us to generate the exact source code by concatenating the generated tokens. The parser is written in a way that it always returns a complete syntax tree. Error conditions are represented as nodes in the tree. This simplifies the code and allows it to be used as an early stage parser, e.g for parsing JS documents in an IDE while editing the source code.:
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-16 17:56:49 +0000