View source for Heuristics for detecting class text encoding

← Heuristics for detecting class text encoding

You do not have permission to edit this page, for the following reasons:

The action you have requested is limited to users in one of the groups: Autoconfirmed users, emailconfirmed, editors.
You must confirm your email address before editing pages. Please set and validate your email address through your user preferences.

You can view and copy the source of this page:

[[Category:Unicode]]
When Eiffel Software implement support for class texts written in Unicode, it is important to detect which Unicode encoding scheme is in use. Note that class authors will want to write class text in UTF-8 for certain (support for other encoding schemes is sporadic amongst programmers text editors), UTF-16/UTF-16BE/UTF-16LE probably (especially on Windows systems, or in East Asia), and least likely in UTF-32/UTF-32BE/UTF-32LE. The ECMA standard does not (yet - I have raised the issue) address the matter of encoding schemes. I trust it will allow either all 7 freely, or (better), just UTF-8 (without a BOM), and UTF-16 and UTF-32 (with the aditional requirement of a BOM).

If my suggestion in [http://www.eiffelroom.com/blog/colin_adams/mixing_unicode_and_latin_1_class_texts Mixing Unicode and Latin-1 class texts] is followed, and I hope it is, then the compiler will already know from the cluster definition, whether or not a class is written in a Unicode encoding scheme or not. Therefore confusion with Latin-1 texts does not arise.

If only UTF-8 (without a BOM), and UTF-16 and UTF-32 (with the aditional requirement of a BOM) are allowed, then the heuristic is simple - if it has a BOM, examine the first four bytes - this determines the encoding. If it is not a BOM, then it is UTF-8.

Otherwise, simple heuristics can reliably determine the encoding scheme of the class by reading the first line of the class text as a sequence of octets (e.g. reading it as a STRING_8 and interpreting the `code' of each "character").

The class text can only begin with one of the following:

White space,
indexing,
notes,
class,
deferred,
expanded,
frozen

or a byte-order-mark (have I missed anything? --[[User:Ericb|Ericb]] 19:21, 30 March 2007 (CEST): yes: comments starting with -- ).

So these can all be tested for all seven possible encodings (you only have to read 8 bytes).

Note that the Eiffel Studio editor should save class texts in an encoding scheme according to a user preference. I recommend allowing only UTF-8 (without a BOM), and UTF-16 and UTF-32 (with the aditional requirement of a BOM).

Return to Heuristics for detecting class text encoding.

Retrieved from "https://dev.eiffel.com/Heuristics_for_detecting_class_text_encoding"

View source for Heuristics for detecting class text encoding

Navigation

Development

Wiki

Search

Tools