File Name Handling in EiffelBase - Revision history

Manus: Added exception for FILE and DIRECTORY

2012-11-09T00:02:11Z

Added exception for FILE and DIRECTORY

Manus: Updated Unicode handling with latest changes

2012-11-09T00:00:11Z

Updated Unicode handling with latest changes

Manus: /* The known limitations for Unix */

2012-10-16T17:16:28Z

‎The known limitations for Unix

Alexander Kogtenkov: Fixed typos.

2012-10-16T07:20:46Z

Fixed typos.

Manus: /* The know limitations for Unix */

2012-10-16T06:02:47Z

‎The know limitations for Unix

Manus: /* EiffelStudio 7.2 and later */

2012-10-16T06:02:15Z

‎EiffelStudio 7.2 and later

Manus at 06:01, 16 October 2012

2012-10-16T06:01:21Z

Manus at 05:56, 16 October 2012

2012-10-16T05:56:30Z

Manus at 05:45, 16 October 2012

2012-10-16T05:45:24Z

Manus: Work in progress

2012-10-16T05:40:41Z

Work in progress

New page

File name handling could be simpler if the Unix world hadn't stated that a file name is simply a sequence of bytes. The impact of such statement is that sequence of bytes can be interpreted differently depending on your locale settings. For example a French user looking at the name of a file would see a different name than a Chinese person looking at the exact same file.

On Windows the story is better as all file names are UTF-16 encoded.

But what about Eiffel that provides a multiplatform way to access files?

== STRING_8 ==
Prior to 7.1, the only way to reference the file system was to use STRING_8 instances to represent a path.

On Unix, if the file was Unicode, that did not matter because really it is not Unicode, it is just a sequence of bytes.

On Windows, if the file had a name that contained a code greater than 256, it would work just fine most of the time since the Eiffel runtime used the ANSI version of the Microsoft APIs. However if you would get a path from the Windows Open File dialog, you will be unable to read that file since it would return a STRING_32 instances.

== STRING_32 ==
Eiffel had support for Unicode for quite a few releases but not at the level of the file classes. In 7.2, we have introduced this capabilities which has unfortunately some restrictions for Unix users.

First we can verify that if you provide a STRING_32 path to the file classes, then it will internally convert it to UTF-16 and use the Microsoft APIs to open the file, read the content of a directory, ... Whenever we get a file back from Windows, we convert it from UTF-16 to STRING_32. Now, if you pass a STRING_8 instance, we did not want to break any code, so when you do so, we would convert the STRING_8 instance to UTF-16 the same way that Microsoft does it by using the ANSI code page which is basically dependent on the language of your OS. Then you can query back the STRING_8 instance, or its STRING_32 conversion.

===What problem for Unix users?===
AS mentioned above, the Unix APIs for file manipulation handles a file name as a byte sequence without specifying any encoding. So if I have a STRING_32 instance and want to create a file, I need to choose an encoding, but which one? In 7.2, we have decided that if yo pass a STRING_8 instance, we pass the byte sequence directly to the OS as done prior to 7.2. However if you pass a STRING_32 instance, then we convert it first to UTF-8 before passing it to the OS.

So far so good. The rule becomes the following:
# You create your own files: use STRING_32 and Unicode and your files will be saved in UTF-8.
# You need to read files in path where no encoding is specified: use a STRING_8 sequence to represent the path.

The caveat for Unix is with the DIRECTORY class. How do we interpret the content of a directory listing since we do not know in advance

← Older revision		Revision as of 00:02, 9 November 2012
Line 28:		Line 28:
	In order to not break existing code but still offer users to provide Unicode path as STRING_32, we have also equipped most API to take a READABLE_STRING_GENERAL as argument. Note that when an API takes a READABLE_STRING_GENERAL, it will interpret all strings as if they were a Unicode character sequence, which means that STRING_8 are Unicode character whose code is between 0 and 255, and STRING_32 is the full set.		In order to not break existing code but still offer users to provide Unicode path as STRING_32, we have also equipped most API to take a READABLE_STRING_GENERAL as argument. Note that when an API takes a READABLE_STRING_GENERAL, it will interpret all strings as if they were a Unicode character sequence, which means that STRING_8 are Unicode character whose code is between 0 and 255, and STRING_32 is the full set.

−	~~Existing~~ APIs that used to take STRING_8 are left as is to not break backward compatibility and the STRING_8 sequence is not interpreted as Unicode character sequence but as a sequence of byte as before.	+	The only exception is with FILE and DIRECTORY where STRING_8 are interpreted in the current locale.
		+
		+	Otherwise, existing APIs that used to take STRING_8 are left as is to not break backward compatibility and the STRING_8 sequence is not interpreted as Unicode character sequence but as a sequence of byte as before.

@@ Line 1: / Line 1: @@
 File name handling could be simpler if the Unix world hadn't decided that a file name is simply a sequence of bytes. The impact of such statement is that sequence of bytes can be interpreted differently depending on your locale settings. For example a French user looking at the name of a file would see a different name than a Chinese person looking at the exact same file.
-On Windows the story is better as all file names are UTF-16 encoded.
+On Windows the story is better as all file names are UTF-16 encoded but Windows does not guarantee that it is a valid UTF-16 sequence, which means that not all fill names are representable in a printed manner.
 But what about Eiffel that provides a multiplatform way to access files?
@@ Line 14: / Line 14: @@
 == EiffelStudio 7.2 and later ==
-Eiffel had support for Unicode since EiffelStudio 5.7 but not at the level of the '''FILE''' classes. In 7.2, we have introduced the possibility to refer to a file using a STRING_32 instance with some restrictions which depend on the platform you are running.
+Eiffel had support for Unicode since EiffelStudio 5.7 but not at the level of the '''FILE''' classes. In 7.2, we have introduced the possibility to refer to a file using a PATH or a READABLE_STRING_GENERAL instance. PATH is a new class of EiffelBase.
-=== Normal behavior ===
+=== PATH ===
-In order to not break existing code, whenever you pass a '''STRING_8''' instance as the name for the '''FILE''' classes, we preserve the older behavior. On Windows, the string is converted using the ANSI code page to Unicode, and on Unix it stays the same.
-If you pass a '''STRING_32''' instance as the name, then the behavior changes slightly. On Windows the Unicode string is still converted but this time the conversion is really just an encoding of the Unicode string into UTF-16. On Unix the Unicode string is encoded as a UTF-8 sequence used as the file name.
+As mentioned before the main issue with handling with file names is that they use 1-byte sequence (UNIX) or a 2-byte sequence (Windows). Usually the encoding will be UTF-8 (UNIX) or UTF-16 (Windows) but there is no guarantee.
-=== The known limitations for Unix ===
+When a file name/path contains such mixed encoding, the file name/path is not representable textually. This is where PATH is handy. PATH is immutable making it easy to manipulate and to share.
-* One Unix limitation is that if you have API returning you a Unicode string for a file, it doesn't tell you for sure that the file is actually UTF-8 encoded in the file system, if it was not UTF-8 encoded then you would not be able to access that file this way.
-* The DIRECTORY class provides a way to iterate a directory content. If the content has files with different encodings which are not UTF-8, then we cannot translate them back to STRING_32 without breaking the roundtrip (i.e. using that STRING_32 instance, we should be able to open the corresponding file). What shall we do? The directory listing is still possible but you need to use the version that gives you back STRING_8 instances, that you can then use to create a FILE instance (but no conversion to STRING_32 should intervene in the process).
+All APIs of EiffelBase and other libraries have been updated to also take a PATH instance as argument each time there is a need to access a file.
-* The final limitation is that building paths cannot mix both STRING_32 and STRING_8 instances.
+=== READABLE_STRING_GENERAL ===
-To summarize, on Unix, the rule becomes the following:
+In order to not break existing code but still offer users to provide Unicode path as STRING_32, we have also equipped most API to take a READABLE_STRING_GENERAL as argument. Note that when an API takes a READABLE_STRING_GENERAL, it will interpret all strings as if they were a Unicode character sequence, which means that STRING_8 are Unicode character whose code is between 0 and 255, and STRING_32 is the full set.
-# Create your own files: use STRING_32 and your file names will be saved in UTF-8.
-# Read files in path where no encoding is specified: use a STRING_8 sequence to represent the path.
+Existing APIs that used to take STRING_8 are left as is to not break backward compatibility and the STRING_8 sequence is not interpreted as Unicode character sequence but as a sequence of byte as before.
-# Building paths cannot mix STRING_32 and STRING_8.

@@ Line 29: / Line 29: @@
 To summarize, on Unix, the rule becomes the following:
-# You create your own files: use STRING_32 and Unicode and your file names will be saved in UTF-8.
+# Create your own files: use STRING_32 and your file names will be saved in UTF-8.
-# You need to read files in path where no encoding is specified: use a STRING_8 sequence to represent the path.
+# Read files in path where no encoding is specified: use a STRING_8 sequence to represent the path.
-# Building paths cannot mix STRING_32 or STRING_8.
+# Building paths cannot mix STRING_32 and STRING_8.

@@ Line 14: / Line 14: @@
 == EiffelStudio 7.2 and later ==
-Eiffel had support for Unicode since EiffelStudio 5.7 but not at the level of the '''FILE''' classes. In 7.2, we have introduced the possibility to refer a file using a STRING_32 instance with some restrictions which depend on the platform you are running.
+Eiffel had support for Unicode since EiffelStudio 5.7 but not at the level of the '''FILE''' classes. In 7.2, we have introduced the possibility to refer to a file using a STRING_32 instance with some restrictions which depend on the platform you are running.
 === Normal behavior ===
@@ Line 21: / Line 21: @@
 If you pass a '''STRING_32''' instance as the name, then the behavior changes slightly. On Windows the Unicode string is still converted but this time the conversion is really just an encoding of the Unicode string into UTF-16. On Unix the Unicode string is encoded as a UTF-8 sequence used as the file name.
-=== The know limitations for Unix ===
+=== The known limitations for Unix ===
 * One Unix limitation is that if you have API returning you a Unicode string for a file, it doesn't tell you for sure that the file is actually UTF-8 encoded in the file system, if it was not UTF-8 encoded then you would not be able to access that file this way.

← Older revision		Revision as of 06:02, 16 October 2012
Line 31:		Line 31:
	# You create your own files: use STRING_32 and Unicode and your file names will be saved in UTF-8.		# You create your own files: use STRING_32 and Unicode and your file names will be saved in UTF-8.
	# You need to read files in path where no encoding is specified: use a STRING_8 sequence to represent the path.		# You need to read files in path where no encoding is specified: use a STRING_8 sequence to represent the path.
		+	# Building paths cannot mix STRING_32 or STRING_8.

@@ Line 14: / Line 14: @@
 == EiffelStudio 7.2 and later ==
-Eiffel had support for Unicode since EiffelStudio 5.7 but not at the level of the file classes. In 7.2, we have introduced the possibility to refer a file using a STRING_32 instance with some restrictions which depend on the platform you are running.
+Eiffel had support for Unicode since EiffelStudio 5.7 but not at the level of the '''FILE''' classes. In 7.2, we have introduced the possibility to refer a file using a STRING_32 instance with some restrictions which depend on the platform you are running.
-=== Windows ===
+=== Normal behavior ===
-First we can verify that if you provide a STRING_32 path to the file classes, then it will internally convert it to UTF-16 and use the Microsoft APIs to open the file, read the content of a directory, ... Whenever we get a file back from Windows, we convert it from UTF-16 to STRING_32. Now, if you pass a STRING_8 instance, we did not want to break any code, so when you do so, we would convert the STRING_8 instance to UTF-16 the same way that Microsoft does it by using the ANSI code page which is basically dependent on the language of your OS. Then you can query back the STRING_8 instance, or its STRING_32 conversion.
+In order to not break existing code, whenever you pass a STRING_8 instance as the name for the '''FILE''' classes, we preserve the older behavior. On Windows, the string is converted using the ANSI code page to Unicode, and on Unix it stays the same.
 AS mentioned above, the Unix APIs for file manipulation handles a file name as a byte sequence without specifying any encoding. So if I have a STRING_32 instance and want to create a file, I need to choose an encoding, but which one? In 7.2, we have decided that if yo pass a STRING_8 instance, we pass the byte sequence directly to the OS as done prior to 7.2. However if you pass a STRING_32 instance, then we convert it first to UTF-8 before passing it to the OS.

@@ Line 1: / Line 1: @@
-File name handling could be simpler if the Unix world hadn't stated that a file name is simply a sequence of bytes. The impact of such statement is that sequence of bytes can be interpreted differently depending on your locale settings. For example a French user looking at the name of a file would see a different name than a Chinese person looking at the exact same file.
+File name handling could be simpler if the Unix world hadn't decided that a file name is simply a sequence of bytes. The impact of such statement is that sequence of bytes can be interpreted differently depending on your locale settings. For example a French user looking at the name of a file would see a different name than a Chinese person looking at the exact same file.
 On Windows the story is better as all file names are UTF-16 encoded.
@@ Line 5: / Line 5: @@
 But what about Eiffel that provides a multiplatform way to access files?
-== STRING_8 ==
+== EiffelStudio 7.1 and older ==
-Prior to 7.1, the only way to reference the file system was to use STRING_8 instances to represent a path.
+Prior to 7.1, the only way to reference the file system was to use '''STRING_8''' instances to represent a path.
-On Unix, if the file was Unicode, that did not matter because really it is not Unicode, it is just a sequence of bytes.
+On Unix, if the file was Unicode, that did not matter because really it is not Unicode, it is just a sequence of bytes that can be represented in a STRING_8 instance.
-On Windows, if the file had a name that contained a code greater than 256, it would work just fine most of the time since the Eiffel runtime used the ANSI version of the Microsoft APIs. However if you would get a path from the Windows Open File dialog, you will be unable to read that file since it would return a STRING_32 instances.
+On Windows, if the file had a name that contained a code greater than 256, it would work just fine most of the time since the Eiffel runtime used the ANSI version of the Microsoft APIs. However if you would get a path from the Windows Open File dialog, you will be unable to read that file since it would return a full blown Unicode string as a STRING_32 instance.
-== STRING_32 ==
+== EiffelStudio 7.2 and later ==
-Eiffel had support for Unicode for quite a few releases but not at the level of the file classes. In 7.2, we have introduced this capabilities which has unfortunately some restrictions for Unix users.
+Eiffel had support for Unicode since EiffelStudio 5.7 but not at the level of the file classes. In 7.2, we have introduced the possibility to refer a file using a STRING_32 instance with some restrictions which depend on the platform you are running.
 First we can verify that if you provide a STRING_32 path to the file classes, then it will internally convert it to UTF-16 and use the Microsoft APIs to open the file, read the content of a directory, ... Whenever we get a file back from Windows, we convert it from UTF-16 to STRING_32. Now, if you pass a STRING_8 instance, we did not want to break any code, so when you do so, we would convert the STRING_8 instance to UTF-16 the same way that Microsoft does it by using the ANSI code page which is basically dependent on the language of your OS. Then you can query back the STRING_8 instance, or its STRING_32 conversion.
-===What problem for Unix users?===
+=== Unix ===
 AS mentioned above, the Unix APIs for file manipulation handles a file name as a byte sequence without specifying any encoding. So if I have a STRING_32 instance and want to create a file, I need to choose an encoding, but which one? In 7.2, we have decided that if yo pass a STRING_8 instance, we pass the byte sequence directly to the OS as done prior to 7.2. However if you pass a STRING_32 instance, then we convert it first to UTF-8 before passing it to the OS.