Compiler and encoding: Difference between revisions
Jump to navigation
Jump to search
New page: Category:compiler From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only accept UTF-8 source code, for simplicity and generality. Before source cod... |
No edit summary |
||
| Line 1: | Line 1: | ||
[[Category:compiler]] | [[Category:compiler]] | ||
From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only | From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only accepts UTF-8 source code, for simplicity and generality. Before source code is passed into the core process of parsing, it is preprocessed and converted into UTF-8. | ||
== Internals == | == Internals == | ||
Abstracted syntax tree now stores STRING_8 as UTF-8 data on each node. There are also different features exporting UTF-8, UTF-32 or the written bytes. Here is an example of how a character é is represented in various levels. | |||
Revision as of 11:34, 30 May 2012
From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only accepts UTF-8 source code, for simplicity and generality. Before source code is passed into the core process of parsing, it is preprocessed and converted into UTF-8.
Internals
Abstracted syntax tree now stores STRING_8 as UTF-8 data on each node. There are also different features exporting UTF-8, UTF-32 or the written bytes. Here is an example of how a character é is represented in various levels.