Ohm-Management - Projektarbeit B-ME
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

README.md 7.2KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240
  1. Overview [![Build Status](https://travis-ci.org/lydell/js-tokens.svg?branch=master)](https://travis-ci.org/lydell/js-tokens)
  2. ========
  3. A regex that tokenizes JavaScript.
  4. ```js
  5. var jsTokens = require("js-tokens").default
  6. var jsString = "var foo=opts.foo;\n..."
  7. jsString.match(jsTokens)
  8. // ["var", " ", "foo", "=", "opts", ".", "foo", ";", "\n", ...]
  9. ```
  10. Installation
  11. ============
  12. `npm install js-tokens`
  13. ```js
  14. import jsTokens from "js-tokens"
  15. // or:
  16. var jsTokens = require("js-tokens").default
  17. ```
  18. Usage
  19. =====
  20. ### `jsTokens` ###
  21. A regex with the `g` flag that matches JavaScript tokens.
  22. The regex _always_ matches, even invalid JavaScript and the empty string.
  23. The next match is always directly after the previous.
  24. ### `var token = matchToToken(match)` ###
  25. ```js
  26. import {matchToToken} from "js-tokens"
  27. // or:
  28. var matchToToken = require("js-tokens").matchToToken
  29. ```
  30. Takes a `match` returned by `jsTokens.exec(string)`, and returns a `{type:
  31. String, value: String}` object. The following types are available:
  32. - string
  33. - comment
  34. - regex
  35. - number
  36. - name
  37. - punctuator
  38. - whitespace
  39. - invalid
  40. Multi-line comments and strings also have a `closed` property indicating if the
  41. token was closed or not (see below).
  42. Comments and strings both come in several flavors. To distinguish them, check if
  43. the token starts with `//`, `/*`, `'`, `"` or `` ` ``.
  44. Names are ECMAScript IdentifierNames, that is, including both identifiers and
  45. keywords. You may use [is-keyword-js] to tell them apart.
  46. Whitespace includes both line terminators and other whitespace.
  47. [is-keyword-js]: https://github.com/crissdev/is-keyword-js
  48. ECMAScript support
  49. ==================
  50. The intention is to always support the latest ECMAScript version whose feature
  51. set has been finalized.
  52. If adding support for a newer version requires changes, a new version with a
  53. major verion bump will be released.
  54. Currently, ECMAScript 2018 is supported.
  55. Invalid code handling
  56. =====================
  57. Unterminated strings are still matched as strings. JavaScript strings cannot
  58. contain (unescaped) newlines, so unterminated strings simply end at the end of
  59. the line. Unterminated template strings can contain unescaped newlines, though,
  60. so they go on to the end of input.
  61. Unterminated multi-line comments are also still matched as comments. They
  62. simply go on to the end of the input.
  63. Unterminated regex literals are likely matched as division and whatever is
  64. inside the regex.
  65. Invalid ASCII characters have their own capturing group.
  66. Invalid non-ASCII characters are treated as names, to simplify the matching of
  67. names (except unicode spaces which are treated as whitespace). Note: See also
  68. the [ES2018](#es2018) section.
  69. Regex literals may contain invalid regex syntax. They are still matched as
  70. regex literals. They may also contain repeated regex flags, to keep the regex
  71. simple.
  72. Strings may contain invalid escape sequences.
  73. Limitations
  74. ===========
  75. Tokenizing JavaScript using regexes—in fact, _one single regex_—won’t be
  76. perfect. But that’s not the point either.
  77. You may compare jsTokens with [esprima] by using `esprima-compare.js`.
  78. See `npm run esprima-compare`!
  79. [esprima]: http://esprima.org/
  80. ### Template string interpolation ###
  81. Template strings are matched as single tokens, from the starting `` ` `` to the
  82. ending `` ` ``, including interpolations (whose tokens are not matched
  83. individually).
  84. Matching template string interpolations requires recursive balancing of `{` and
  85. `}`—something that JavaScript regexes cannot do. Only one level of nesting is
  86. supported.
  87. ### Division and regex literals collision ###
  88. Consider this example:
  89. ```js
  90. var g = 9.82
  91. var number = bar / 2/g
  92. var regex = / 2/g
  93. ```
  94. A human can easily understand that in the `number` line we’re dealing with
  95. division, and in the `regex` line we’re dealing with a regex literal. How come?
  96. Because humans can look at the whole code to put the `/` characters in context.
  97. A JavaScript regex cannot. It only sees forwards. (Well, ES2018 regexes can also
  98. look backwards. See the [ES2018](#es2018) section).
  99. When the `jsTokens` regex scans throught the above, it will see the following
  100. at the end of both the `number` and `regex` rows:
  101. ```js
  102. / 2/g
  103. ```
  104. It is then impossible to know if that is a regex literal, or part of an
  105. expression dealing with division.
  106. Here is a similar case:
  107. ```js
  108. foo /= 2/g
  109. foo(/= 2/g)
  110. ```
  111. The first line divides the `foo` variable with `2/g`. The second line calls the
  112. `foo` function with the regex literal `/= 2/g`. Again, since `jsTokens` only
  113. sees forwards, it cannot tell the two cases apart.
  114. There are some cases where we _can_ tell division and regex literals apart,
  115. though.
  116. First off, we have the simple cases where there’s only one slash in the line:
  117. ```js
  118. var foo = 2/g
  119. foo /= 2
  120. ```
  121. Regex literals cannot contain newlines, so the above cases are correctly
  122. identified as division. Things are only problematic when there are more than
  123. one non-comment slash in a single line.
  124. Secondly, not every character is a valid regex flag.
  125. ```js
  126. var number = bar / 2/e
  127. ```
  128. The above example is also correctly identified as division, because `e` is not a
  129. valid regex flag. I initially wanted to future-proof by allowing `[a-zA-Z]*`
  130. (any letter) as flags, but it is not worth it since it increases the amount of
  131. ambigous cases. So only the standard `g`, `m`, `i`, `y` and `u` flags are
  132. allowed. This means that the above example will be identified as division as
  133. long as you don’t rename the `e` variable to some permutation of `gmiyus` 1 to 6
  134. characters long.
  135. Lastly, we can look _forward_ for information.
  136. - If the token following what looks like a regex literal is not valid after a
  137. regex literal, but is valid in a division expression, then the regex literal
  138. is treated as division instead. For example, a flagless regex cannot be
  139. followed by a string, number or name, but all of those three can be the
  140. denominator of a division.
  141. - Generally, if what looks like a regex literal is followed by an operator, the
  142. regex literal is treated as division instead. This is because regexes are
  143. seldomly used with operators (such as `+`, `*`, `&&` and `==`), but division
  144. could likely be part of such an expression.
  145. Please consult the regex source and the test cases for precise information on
  146. when regex or division is matched (should you need to know). In short, you
  147. could sum it up as:
  148. If the end of a statement looks like a regex literal (even if it isn’t), it
  149. will be treated as one. Otherwise it should work as expected (if you write sane
  150. code).
  151. ### ES2018 ###
  152. ES2018 added some nice regex improvements to the language.
  153. - [Unicode property escapes] should allow telling names and invalid non-ASCII
  154. characters apart without blowing up the regex size.
  155. - [Lookbehind assertions] should allow matching telling division and regex
  156. literals apart in more cases.
  157. - [Named capture groups] might simplify some things.
  158. These things would be nice to do, but are not critical. They probably have to
  159. wait until the oldest maintained Node.js LTS release supports those features.
  160. [Unicode property escapes]: http://2ality.com/2017/07/regexp-unicode-property-escapes.html
  161. [Lookbehind assertions]: http://2ality.com/2017/05/regexp-lookbehind-assertions.html
  162. [Named capture groups]: http://2ality.com/2017/05/regexp-named-capture-groups.html
  163. License
  164. =======
  165. [MIT](LICENSE).