Design of Roundtrip Parser - Revision history

Jasonw: /* How to regenerate code */

2007-08-09T08:49:25Z

How to regenerate code

← Older revision		Revision as of 08:49, 9 August 2007
Line 242:		Line 242:


	Suppose that the condition expression node has been processed, which means that the expression sub node has been visited and the the text of the condition expression has been printed out, and the item at position 102 in the match list has been processed as well. The next sub node to visit is the '''then''' keyword node. And item at position 104 in the match list is associated with this '''then''' keyword node. But before we process this node, we check if in the match list, there are unprocessed items before position 104. And in our example, item at position 103 is not processed. So we process that item first~~, meaning~~ printing out the break token directly. And then we continue to process the '''then''' keyword node.		Suppose that the condition expression node has been processed, which means that the expression sub node has been visited and the the text of the condition expression has been printed out, and the item at position 102 in the match list has been processed as well. The next sub node to visit is the '''then''' keyword node. And item at position 104 in the match list is associated with this '''then''' keyword node. But before we process this node, we check if in the match list, there are unprocessed items before position 104. And in our example, item at position 103 is not processed. So we process that item first by printing out the break token directly. And then we continue to process the '''then''' keyword node.

	One thing to note is that after processing the last '''end''' keyword node in a class AST node, we also need to check if there are still items left in the match list because end class comments and breaks can appear after the last '''end''' keyword. Consider the following code:		One thing to note is that after processing the last '''end''' keyword node in a class AST node, we also need to check if there are still items left in the match list because end class comments and breaks can appear after the last '''end''' keyword. Consider the following code:

Jasonw: /* How to regenerate code */

2007-08-09T08:48:49Z

How to regenerate code

← Older revision		Revision as of 08:48, 9 August 2007
Line 242:		Line 242:


	Suppose that the condition expression node has been processed, which means that the expression sub node has been visited and the the text of the condition expression has been printed out, and the item at position 102 in the match list has been processed as well. The next sub node to visit is the ''then'' keyword node. And item at position 104 in the match list is associated with this ''then'' keyword node. But before we process this node, we check if in the match list, there are unprocessed items before position 104. And in our example, item at position 103 is not processed. So we process that item first, meaning printing out the break token directly. And then we continue to process the ''then'' keyword node.		Suppose that the condition expression node has been processed, which means that the expression sub node has been visited and the the text of the condition expression has been printed out, and the item at position 102 in the match list has been processed as well. The next sub node to visit is the '''then''' keyword node. And item at position 104 in the match list is associated with this '''then''' keyword node. But before we process this node, we check if in the match list, there are unprocessed items before position 104. And in our example, item at position 103 is not processed. So we process that item first, meaning printing out the break token directly. And then we continue to process the '''then''' keyword node.

	One thing to note is that after processing the last ''end'' keyword node in a class AST node, we also need to check if there are still items left in the match list because end class comments and breaks can appear after the last ''end'' keyword. Consider the following code:		One thing to note is that after processing the last '''end''' keyword node in a class AST node, we also need to check if there are still items left in the match list because end class comments and breaks can appear after the last '''end''' keyword. Consider the following code:

	<code>[eiffel, N]		<code>[eiffel, N]
Line 254:		Line 254:
	</code>		</code>

	The last comment section needs to be taken care of after processing the ''end'' keyword node.		The last comment section needs to be taken care of after processing the '''end''' keyword node.

	An quick way to generate source code is by just printing out every item in the match list. This is suitable in cases that no code transformation is needed.		An quick way to generate source code is by just printing out every item in the match list. This is suitable in cases that no code transformation is needed.

	That's how the original code is generated.		That's how the original code is generated.

Jasonw: /* How to regenerate code */

2007-08-09T08:47:25Z

How to regenerate code

← Older revision		Revision as of 08:47, 9 August 2007
Line 242:		Line 242:


	Suppose that the condition expression node has been processed, which means that the expression sub node has been visited and the the text of the condition expression has been printed out, as the item at position 102 in the match list has been processed as well. The next sub node to visit is the ''then'' keyword node. And item at position 104 in the match list is associated with this ''then'' keyword node. But before we process this node, we check if in the match list, there are unprocessed items before position 104. And in our example, item at position 103 is not processed. So we process that item first, meaning printing out the break token directly. And then we continue to process the ''then'' keyword node.		Suppose that the condition expression node has been processed, which means that the expression sub node has been visited and the the text of the condition expression has been printed out, and the item at position 102 in the match list has been processed as well. The next sub node to visit is the ''then'' keyword node. And item at position 104 in the match list is associated with this ''then'' keyword node. But before we process this node, we check if in the match list, there are unprocessed items before position 104. And in our example, item at position 103 is not processed. So we process that item first, meaning printing out the break token directly. And then we continue to process the ''then'' keyword node.

	One thing to note is that after processing the last ''end'' keyword node in a class AST node, we also need to check if there are still items left in the match list because end class comments and breaks can appear after the last ''end'' keyword. Consider the following code:		One thing to note is that after processing the last ''end'' keyword node in a class AST node, we also need to check if there are still items left in the match list because end class comments and breaks can appear after the last ''end'' keyword. Consider the following code:
Line 255:		Line 255:

	The last comment section needs to be taken care of after processing the ''end'' keyword node.		The last comment section needs to be taken care of after processing the ''end'' keyword node.

			An quick way to generate source code is by just printing out every item in the match list. This is suitable in cases that no code transformation is needed.

	That's how the original code is generated.		That's how the original code is generated.

Jasonw: /* How to regenerate code */

2007-08-09T08:43:26Z

How to regenerate code

← Older revision		Revision as of 08:43, 9 August 2007
Line 228:		Line 228:

	== How to regenerate code ==		== How to regenerate code ==
	To regenerate code from AST nodes and associated match list, we need to use a visitor. This visitor will recursively process the AST tree with the help of the match list and index information stored in every terminal AST node. From the AST node, we can arrive every terminal tokens originally in the source code except for break tokens. So for non-break tokens, we use the index stored in AST nodes to get the original text of that token from the match list, and then we just print out the text. But before we print the text of a non-break token, we need to check in the match list if before current non-break token, there is some break tokens that are not processed. If so, we print out those break tokens first.		To regenerate code from AST nodes and the associated match list, we need to use a visitor. This visitor will recursively process the AST tree with the help of the match list and index information stored in every terminal AST node. From the AST node, we can arrive every terminal tokens originally in the source code except for break tokens. So for non-break tokens, we use the index stored in AST nodes to get the original text of that token from the match list, and then we just print out the text. But before we print the text of a non-break token, we need to check in the match list if before current non-break token, there is some break tokens that are not processed. If so, we print out those break tokens first.

Jasonw: /* Switch between roundtrip and non-roundtrip mode */

2007-08-09T08:41:42Z

Switch between roundtrip and non-roundtrip mode

← Older revision		Revision as of 08:41, 9 August 2007
Line 210:		Line 210:


	The ~~major~~ performance penalty (compared with non roundtrip parser) are:		The performance penalty (compared with non roundtrip parser) are:

	(1) creating of AST nodes to represent keywords		(1) creating of AST nodes to represent keywords

Jasonw: /* Switch between roundtrip and non-roundtrip mode */

2007-08-09T08:40:21Z

Switch between roundtrip and non-roundtrip mode

← Older revision		Revision as of 08:40, 9 August 2007
Line 200:		Line 200:

	=== Switch between roundtrip and non-roundtrip mode ===		=== Switch between roundtrip and non-roundtrip mode ===
	The involvement of the roundtrip function will definitely slow down the parsing process because we more time is spent in creating new objects, copying text and maintaining the match list. To minimize the performance overhead, the Eiffel parser is designed to be able to work on either roundtrip mode or non-roundtrip mode.		The involvement of the roundtrip function will definitely slow down the parsing process because more time is spent in creating new objects, copying text and maintaining the match list. To minimize the performance overhead, the Eiffel parser is designed to be able to work on either roundtrip mode or non-roundtrip mode.

Jasonw: /* Staff match list */

2007-08-09T08:39:10Z

Staff match list

← Older revision		Revision as of 08:39, 9 August 2007
Line 193:		Line 193:
	</code>		</code>

	For keywords, similar code patterns ~~presents~~.		For keywords, similar code patterns present.

	We need to do some extra work when matching reserved word once because there are regular expressions embedded in the expression to match once.		We need to do some extra work when matching reserved word once because there are regular expressions embedded in the expression to match once.

Jasonw: /* Staff match list */

2007-08-09T08:30:42Z

Staff match list

← Older revision		Revision as of 08:30, 9 August 2007
Line 159:		Line 159:
	[[image:if_clause_ast.jpg]]		[[image:if_clause_ast.jpg]]

	From the above, we know that '''if''' keyword is at position 100 in the match list, and the space break after '''if''' keyword is at position 101, and so on. Here we assume that the condition expression is only comprised of one terminal token, so it's at position 102. The compound part if the ''if'' statement can contain many terminal tokens, so it's marked as 106~n.		From the above, we know that '''if''' keyword is at position 100 in the match list, and the space break after '''if''' keyword is at position 101, and so on. Here we assume that the condition expression is only comprised of one terminal token, so it's at position 102. The compound part of the ''if'' statement can contain many terminal tokens, so it's marked as 106~n.

	When the above AST for ''if'' statement is visited in flavor of code regeneration, we know that the original text of the ''if'' keyword can be retrieved from the match list at position 100.		When the above AST for ''if'' statement is visited in flavor of code regeneration, we know that the original text of the ''if'' keyword can be retrieved from the match list at position 100.

Jasonw: /* Implementation details */

2007-08-09T08:29:03Z

Implementation details

← Older revision		Revision as of 08:29, 9 August 2007
Line 99:		Line 99:

	== Implementation details ==		== Implementation details ==
	To implement the roundtrip function described above, we need to modify AST nodes and Eiffel scanner/parser ~~such that~~:		To implement the roundtrip function described above, we need to modify AST nodes and Eiffel scanner/parser in the following way:
	* ~~Added~~ nodes representing keywords in existing AST nodes		* Add nodes representing keywords in existing AST nodes
	* For every token recognized by the scanner, we insert this token (it can be either a normal token or a break token) into the match list. If it's a normal token, we attach the index of it in the match list to the related AST node.		* For every token recognized by the scanner, we insert this token (it can be either a normal token or a break token) into the match list. If it's a normal token, we attach the index of it in the match list to the related AST node.

Jasonw: /* Implementation details */

2007-08-09T08:27:45Z

Implementation details

← Older revision		Revision as of 08:27, 9 August 2007
Line 102:		Line 102:
	* Added nodes representing keywords in existing AST nodes		* Added nodes representing keywords in existing AST nodes
	* For every token recognized by the scanner, we insert this token (it can be either a normal token or a break token) into the match list. If it's a normal token, we attach the index of it in the match list to the related AST node.		* For every token recognized by the scanner, we insert this token (it can be either a normal token or a break token) into the match list. If it's a normal token, we attach the index of it in the match list to the related AST node.


	=== Add AST nodes for keywords ===		=== Add AST nodes for keywords ===