Compiler and encoding - Revision history

Manus: /* Data Storage */

2012-06-08T05:04:57Z

‎Data Storage

Ted: /* Validility */

2012-05-31T04:19:27Z

‎Validility

Ted: /* Data Storage */

2012-05-31T04:16:35Z

‎Data Storage

Ted: /* Validility */

2012-05-31T03:29:30Z

‎Validility

Ted: /* Validility */

2012-05-31T03:27:40Z

‎Validility

Ted: /* Internals */

2012-05-31T03:11:00Z

‎Internals

Ted: /* Internals */

2012-05-30T12:04:58Z

‎Internals

Ted: /* Internals */

2012-05-30T12:03:09Z

‎Internals

Ted at 11:34, 30 May 2012

2012-05-30T11:34:41Z

Ted: New page: Category:compiler From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only accept UTF-8 source code, for simplicity and generality. Before source cod...

2012-05-30T11:00:51Z

New page: Category:compiler From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only accept UTF-8 source code, for simplicity and generality. Before source cod...

New page

[[Category:compiler]]

From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only accept UTF-8 source code, for simplicity and generality. Before source code is passed into the core process of parsing, it is preprocessed and converted into UTF-8.

== Internals ==

@@ Line 56: / Line 56: @@
 <span lang="EN-US">5. Runtime</span>
 | style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" |
-<span lang="EN-US">0xC3A9 (STRING_8)</span>
+<span lang="EN-US">0xE9 (STRING_8)</span>
 <span lang="EN-US">0xE9 (STRING_32)</span>

@@ Line 83: / Line 83: @@
 |-
 | STRING_8 manifest
-| Only Standard ASCII is valid (0-127)
+| Unicode point (0-255)?
 | Valid (taken as bytes)
 |-

@@ Line 56: / Line 56: @@
 <span lang="EN-US">5. Runtime</span>
 | style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" |
-<span lang="EN-US">0xC3A9 (STRING_8 Rejected)</span>
+<span lang="EN-US">0xC3A9 (STRING_8)</span>
 <span lang="EN-US">0xE9 (STRING_32)</span>

@@ Line 77: / Line 77: @@
 The following table shows how manifest strings are validated by the compiler:
-{|
+{| class="MsoTableGrid" style="width: 431.25pt; border-collapse: collapse; border: none" width="575" border="1"
+|
 | Explicit Encoding

← Older revision		Revision as of 03:27, 31 May 2012
Line 75:		Line 75:
	* Implicit		* Implicit
	** Implicit encoding is taken as ISO-8859-1 for compatibility, if no source code encoding is specified.		** Implicit encoding is taken as ISO-8859-1 for compatibility, if no source code encoding is specified.
		+
		+	The following table shows how manifest strings are validated by the compiler:
		+	{\|
		+	\|
		+	\| Explicit Encoding
		+	\| Implicit Encoding (ISO-8859-1)
		+	\|-
		+	\| STRING_8 manifest
		+	\| Only Standard ASCII is valid (0-127)
		+	\| Valid (taken as bytes)
		+	\|-
		+	\| STRING_32 manifest
		+	\| Valid
		+	\| Valid
		+	\|}

@@ Line 4: / Line 4: @@
 == Internals ==
 Abstracted syntax tree now stores STRING_8 as UTF-8 data on each node. There are also different features exporting UTF-8, UTF-32 or the written bytes.
@@ Line 65: / Line 66: @@
 </div>

← Older revision		Revision as of 12:03, 30 May 2012
Line 5:		Line 5:
	== Internals ==		== Internals ==
	Abstracted syntax tree now stores STRING_8 as UTF-8 data on each node. There are also different features exporting UTF-8, UTF-32 or the written bytes. Here is an example of how a character é is represented in various levels.		Abstracted syntax tree now stores STRING_8 as UTF-8 data on each node. There are also different features exporting UTF-8, UTF-32 or the written bytes. Here is an example of how a character é is represented in various levels.
		+
		+	<div class="WordSection1" style="layout-grid: 15.6pt">
		+
		+	{\| class="MsoTableGrid" style="width: 431.25pt; border-collapse: collapse; border: none" width="575" border="1"
		+	\| style="width: 147.15pt; border: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="196" valign="top" \|
		+	<span lang="EN-US">Source encoding</span>
		+	\| style="width: 142.05pt; border: solid windowtext 1.0pt; border-left: none; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">UTF-8 (BOM)</span>
		+	\| style="width: 142.05pt; border: solid windowtext 1.0pt; border-left: none; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">ISO-8859-1</span>
		+	\|-
		+	\| style="width: 147.15pt; border: solid windowtext 1.0pt; border-top: none; padding: 0cm 5.4pt 0cm 5.4pt" width="196" valign="top" \|
		+	<span lang="EN-US">Bytes in source</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xC3A9</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xE9</span>
		+	\|-
		+	\| style="width: 147.15pt; border: solid windowtext 1.0pt; border-top: none; padding: 0cm 5.4pt 0cm 5.4pt" width="196" valign="top" \|
		+	<span lang="EN-US">1. {STRING_AS}.value</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xC3A9</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xC3A9</span>
		+	\|-
		+	\| style="width: 147.15pt; border: solid windowtext 1.0pt; border-top: none; padding: 0cm 5.4pt 0cm 5.4pt" width="196" valign="top" \|
		+	<span lang="EN-US">2. {STRING_AS}.binary_value</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xC3A9</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xE9</span>
		+	\|-
		+	\| style="width: 147.15pt; border: solid windowtext 1.0pt; border-top: none; padding: 0cm 5.4pt 0cm 5.4pt" width="196" valign="top" \|
		+	<span lang="EN-US">3. {STRING_AS}.value_32</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xE9</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xE9</span>
		+	\|-
		+	\| style="width: 147.15pt; border: solid windowtext 1.0pt; border-top: none; padding: 0cm 5.4pt 0cm 5.4pt" width="196" valign="top" \|
		+	<span lang="EN-US">4. {STRING_AS}.string_value_32</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xE9</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xE9</span>
		+	\|-
		+	\| style="width: 147.15pt; border: solid windowtext 1.0pt; border-top: none; padding: 0cm 5.4pt 0cm 5.4pt" width="196" valign="top" \|
		+	<span lang="EN-US">5. Runtime</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xC3A9 (STRING_8 Rejected)</span>
		+
		+	<span lang="EN-US">0xE9 (STRING_32)</span>
		+	\| style="width: 142.05pt; border-top: none; border-left: none; border-bottom: solid windowtext 1.0pt; border-right: solid windowtext 1.0pt; padding: 0cm 5.4pt 0cm 5.4pt" width="189" valign="top" \|
		+	<span lang="EN-US">0xE9 (STRING_8)</span>
		+
		+	<span lang="EN-US">0xE9 (STRING_32)</span>
		+	\|}
		+
		+	</div>

@@ Line 1: / Line 1: @@
 [[Category:compiler]]
-From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only accept UTF-8 source code, for simplicity and generality. Before source code is passed into the core process of parsing, it is preprocessed and converted into UTF-8.
+From 6.7, the compiler has been equiped with a Unicode parser. The core of the parser only accepts UTF-8 source code, for simplicity and generality. Before source code is passed into the core process of parsing, it is preprocessed and converted into UTF-8.
 == Internals ==