EiffelStudio: an EiffelSoftware project - User contributions [en]

Internationalization/User guide

2006-10-14T06:33:04Z

Etienner: /* Usage */ small correction

==Overview==

The [[i18n]] library is intended to enable localisation of Eiffel programs.

''localisation'' is the process of adapting a piece of software to a specific place - the ''locale'', often expressed as a combination of language and country codes.

This normally means not only displaying strings in the appropriate language, but also adapting number formatting, date and time formatting etc. to use local conventions.
The i18n library provides formatting facilities for numbers, currency values and dates, and the ability to identify and load translated strings at run-time.

==Interface==

The library provides most of it's services through one class: LOCALE. This presents all formatting and translation facilities for a given locale.
LOCALE objects can't be created directly: one must go though the LOCALE_MANAGER class. A LOCALE_MANAGER finds out what information for which locales is available, and offers a list to chose from. It will then load the information for the chosen locale into a LOCALE object and give it to you. With this LOCALE object you can do the really interesting things, like formatting dates and translating strings.

===Choosing a locale===

First you must have a LOCALE_MANAGER. For details on creating them, please see the '''Datasources''' section.
If you just want to use whatever locale is the default on the user's machine, as in most cases, then it's easy to get a LOCALE object: just call get_system_locale.
<code>[eiffel, N]
my_locale := locale_mananger.get_system_locale
</code>

If you want a specific locale, it's going to be a bit more complicated.
A LOCALE_MANAGER knows what locales are available and exactly what information is available for a specific locale.
You can get a list of all locales that are available to some degree by calling the available_locales feature.
A locale is identified by a LOCALE_ID object, and normally this has two components: language code and country code.
For example, the US english locale has language code "en" and country code "US".
If you've found a locale id that you like in the list returned by available_locales, you can check exactly what is vailable for it by using
<code>[eiffel, N]
has_translations (a_locale_id: I18N_LOCALE_ID): BOOLEAN
has_formatting_info (a_locale_id: I18N_LOCALE_ID): BOOLEAN
</code>

You can then retrieve the corresponding LOCALE object by calling get_locale.

===Using your locale===

Now you've got a LOCALE. After the initial burst of euphoria has faded away, what can you do with it?

====String translation====
=====Interface=====
<code>[eiffel, N]
translate (original: STRING_GENERAL): STRING_32
translate_plural (original_singular, original_plural: STRING_GENERAL; plural_form : INTEGER): STRING_32
format_string (original: STRING_GENERAL; token_values: TUPLE[STRING_GENERAL]): STRING_32
</code>
=====Usage=====
In naïvely written software, you can often spot things like
<code>[eiffel, N]
io.put_string("My hovercraft is full of eels")
</code>
We'll use this example to illustrate the use of the string translation features of the i18n library.

If, as above, there is just one constant string to translate, the solution is very easy: simply use the translate function.
The resulting code would look like this
<code>[eiffel, N]
io.put_string(my_locale.translate("My hovercraft is full of eels"))
</code>
If the translate function can't find a translation for this string it will simply return the original string - better then nothing!

But life is, of course, not always that simple. What if we have to deal with plurals? The "traditional" way of doing this is something like:
<code>[eiffel, N]
n := get_number_of_hovercraft
if n = 1 then
io.put_string("My hovercraft is full of eels")
else
io.put_string("My hovercraft are full of eels")
end
</code>

This is not so easy to translate as the above. Why can't we just translate both strings?

Depending on the language, there may be up to 4 different types of plural forms, used in strange and exotic ways. Clearly, it is important to know exactly _how_ many hovercraft there are so that we can choose the right plural form. This can be done by the translate_plural function, which we can use in this way:
<code>[eiffel,n]
n := get_number_of_hovercraft
io.put_string(my_locale.translate_plural("My hovercraft is full of eels","My hovercraft are full of eels",n))
</code>
This function will choose and return a translation in the correct plural form. If it can't find one, it will behave like translate and return either the original singular string or the original plural string, following English grammatical rules.

Often even the above is not enough. What if you want to tell the world exactly how many hovercraft you have? You might write something like this:
<code>[eiffel, N]
n := get_number_of_hovercraft
if n = 1 then
io.put_string("My hovercraft is full of eels")
else
io.put_string("My "+n.out+" hovercraft are full of eels")
end
</code>
How can translate_plural handle this? It needs some reinforcements: the solution is to also use ''string templates''.
This means that we can embed codes like "$1" in a string and replace them in the translation by the actual values.
Let's see how this works:
<code>[eiffel,n]
n := get_number_of_hovercraft
plural_string := my_locale.translate_plural("My hovercraft is full of eels","My $1 hovercraft are full of eels",n)
io.put_string(my_locale.format_string(plural_string, [n]))
</code>
To replace the ''escape codes'', such as $1, $2, we use the function format_string. This replaces all the escape codes it finds by the values in a tuple that you give to it a an argument.

====Formatting====
=====Interface=====
TODO!
=====Usage=====
TODO!

==String extraction==

Somebody has to translate all these strings. Mostly, this isn't a programmer, so somehow you have to be able to give this translator a list of strings that you want translated.

We can extract these strings from your application fairly easily by simply looking at the arguments for each call to translate or translate_plural. By clicking on a handy button in EiffelStudio, these strings will be extracted and placed in a [http://www.gnu.org/software/gettext/manual/html_node/gettext_9.html#SEC9 .po file].

TODO: add more details about clicky thing.

A .po file is a reasonably widespread format for storing strings to be translated. There are several tools to aid translation of these files, such as [http://www.poedit.org/ poEdit] for Windows and [http://kbabel.kde.org/ KBabel] or [http://gtranslator.sourceforge.net/ gtranslator] for KDE and Gnome.
There are also many tools to convert .po files to other formats, such as the xml [http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xliff xliff format].

==Datasources==

The library has to load the translated strings from somewhere - sadly we can't do on the fly translation but if you can, please tell us!
Instead we load the strings from a datasource. This is appropriately generic: it could be anything, from a database to a system that queries a server via RPC or SOAP, but currently we only have one implementation: files. And in fact, we only support one type of file: the .mo file format.
The library can't guess the type of datasource you want to use, so you have to tell it when you create a LOCALE_MANAGER.
This is done via an ''uri''. Currently all uris are interpreted as directories where string catalog files may be found, and any .mo files in this directory will be used by the i18n library. This means that creating a LOCALE_MANAGER looks somewhat like this:
<code>[eiffel, N]
create my_locale_mananger.make("/path/to/my/files")
</code>

===Mo files===

The mo file format is defined by the GNU [http://www.gnu.org/software/gettext/ gettext] library, a widely-used C library that allows localisation of text strings. We support UTF-8 encoded mo files.

How do you get these .mo files? Once your translator has finished translating a .po file, you can convert it into a .mo file by using the ''msgfmt'' tool, which is obtainable as part of the gettext package under unix and distributed with poEdit under Windows.
The resulting mo file should be named with the locale identifier of the locale it is intended for (zh_CN.mo or de_CH.mo, for example) and placed in the appropriate directory - this is to say, the one that you give to LOCALE_MANAGER as an uri.

The .mo file should then be seen and used by the i18n library.

Talk:Main Page

2006-09-17T16:40:43Z

Etienner: remove vandalism...

Talk:Main Page

2006-09-15T11:38:11Z

Etienner: remove vandalism...

User talk:Manus

2006-09-12T22:18:33Z

Etienner: /* Vandalism */ again

==Vandalism==

Hi Manus, I wanted to inform you that there are two users that did some vandalism:
* [[User:Ceas]] did [http://eiffelsoftware.origo.ethz.ch/index.php?title=Main_Page&diff=next&oldid=4215 this]
* [[User:Moralest]] did [http://eiffelsoftware.origo.ethz.ch/index.php?title=Main_Page&diff=4442&oldid=4401 this]
* [[B.Gates]] did [http://eiffelsoftware.ethz.ch/index.php?title=Main_Page&diff=4545&oldid=4529 this]
I propose to block their username... I've already removed the changes. [[User:Etienner|'''E''''''T''''''N''']] [[User talk:Etienner|'''talk''']]

: Not worth it; those are certainlyjust bots, probably run from some zombie. [[User:Leo|Leo]] 10:13, 4 September 2006 (CEST)

: Or they were users with weak passwords, if that is possible (I currently do not know what our password policy is) [[User:Schoelle|Bernd]]

Main Page

2006-09-12T22:16:43Z

Etienner: revert

Welcome to [http://www.eiffel.com/ Eiffel Software]'s Wiki, the home page for Eiffel Software's Open Source initiative in collaboration with the [http://se.inf.ethz.ch/ Chair of Software Engineering] at [http://www.ethz.ch ETH Zurich].

{{block|'''Project ideas''' 

If you'd like to contribute to the progress of Eiffel technology but don't know where to start, be sure to
take a look at the [[:Category:Projects| project suggestion pages]]. They contain lots of great ideas for projects large and small!}}

To download or look at the source code of EiffelStudio, please go to the [[Repository| SVN repository]] page.

To compile the source code, you will need a snapshot of the current development release of EiffelStudio. To get it go to the [[Downloads| Downloads]] page.

=== Wiki Contents ===

* [[:Category:EiffelStudio|EiffelStudio]]
* [[:Category:EiffelBuild|EiffelBuild]]
* [[:Category:Compiler| Eiffel Compiler]]
* [[:Category:Runtime| Eiffel Runtime]]
* [[:Category:Library| Eiffel Libraries]]:
** [[:Category:EiffelBase| EiffelBase]]
** [[:Category:EiffelVision2| EiffelVision2]]
** [[:Category:WEL| WEL]]
* [[:Category:Eiffel for .NET| Eiffel for .NET]]
* [[:Category:Documentation| Documentation]]
* [[Roadmap|EiffelStudio Roadmap]]
* [[Repository| SVN repository]]
* [[Contributing| How To Contribute?]]
* [[:Category:Projects| Projects and project suggestions]]
* [[URLs| Useful URLs]]
* [[TODOs| Project TODOs]]
* [[Tools| Tools]]

Internationalization/dotnet locale

2006-09-07T11:34:51Z

Etienner: /* Retrieving locale information */

[[Category:Internationalization]]

== Summary ==
There is a class named CULTURE_INFO in "mscorlib" in EiffelStudio, which provides information about a specific culture such as:
* associated language
* sublanguage
* country/region
* calendar
* cultural conventions

== Listing available locales ==

I didn't find a function that returns all the available locale informations, I think because under .NET, they are all avaiable.

== Checking for default locale ==

<code>[eiffel,N]
a_culture_info : CULTURE_INFO
a_culture_info.current_culture.name : SYSTEM_STRING
-- returns the name of the current culture info, (in the format described below).
</code>

== Retrieving locale information ==
To obtain informations on a specific locale, it suffices to create an instance of CULTURE_INFO. There are basically two ways to do this:

# Create it with the culture identifier. This identifier maps every culture/language to an integer. It is not very usefull, because I did not find a logic behind the mapping function... You can find a table on [http://msdn2.microsoft.com/en-us/library/ms172470.aspx here]
# Create it with the name (STRING) of the culture. The culture names follow the RFC 1766 standard in the format "<languagecode2>-<country/regioncode2>", where <languagecode2> is a lowercase two-letter code derived from ISO 639-1 and <country/regioncode2> is an uppercase two-letter code derived from ISO 3166.

You can find a detailed description of RFC 1766

The CULTURE_INFO Class has some attibutes useful for the retrieving of the informations about Date, Time and Number formatting, they are:

* date_time_format: DATE_TIME_FORMAT_INFO ([http://msdn2.microsoft.com/en-us/library/system.globalization.datetimeformatinfo.aspx see here])

{| border="1"
|+ Some information that is available in DATE_TIME_FORMAT_INFO
! Information !! Result for locale "it-CH"
|-
| full_date_time_pattern || dddd, d. MMMM yyyy HH:mm:ss
|-
| long_date_pattern || dddd, d. MMMM yyyydddd
|-
| long_time_pattern || HH:mm:ss
|-
| month_day_pattern || d. MMMM
|-
| rfc1123_pattern || ddd, dd MMM yyyy HH':'mm':'ss 'GMT'
|-
| pm_designator ||
|-
| short_date_pattern || dd.MM.yyyy
|-
| short_time_pattern || HH:mm
|-
| sortable_date_time_pattern || yyyy'-'MM'-'dd'T'HH':'mm':'ss
|-
| time_separator ||
|}

where:
:*d, %d: The day of the month. Single-digit days will not have a leading zero. Specify "%d" if the format pattern is not combined with other format patterns.
:*dd: The day of the month. Single-digit days will have a leading zero.
:*ddd: The abbreviated name of the day of the week, as defined in AbbreviatedDayNames.
:*dddd: The full name of the day of the week, as defined in DayNames.
:*HH: The hour in a 24-hour clock. Single-digit hours will have a leading zero.
:*mm: The minute. Single-digit minutes will have a leading zero.
:*ss: the second. Single-digit seconds will have a leading zero.
:*yyyy: The year in four or five digits (depending on the calendar used), including the century. Will pad with leading zeroes to get four digits. Thai Buddhist and Korean calendars both have five digit years; users selecting the "yyyy" pattern will see all five digits without leading zeros for calendars that have five digits. Exception: the Japanese and Taiwan calendars always behave as if "yy" was selected.



* there are also functions that return abbreviations of names of months and day:
** get_abbreviated_day_name (dayofweek: DAY_OF_WEEK) : SYSTEM_STRING
** get_abbreviated_month_name (month: INTEGER) : SYSTEM_STRING

* number_format: NUMBER_FORMAT_INFO ([http://msdn2.microsoft.com/en-us/library/system.globalization.numberformatinfo.aspx see here])

{| border="1"
|+ Some information that is available in NUMBER_FORMAT_INFO
! Information !! Result for locale "it-CH"
|-
| currency_group_separator || '
|-
| currency_symbol || SFr.
|-
| currency_decimal_digits || 2
|-
| negative_sign || -
|-
| number_decimal_digits || 2
|-
| number_decimal_separator || .
|-
| number_group_separator || '
|-
| positive_sign || +
|-
| negative_sign || -
|-
| negative_infinity_symbol || -Infinito
|-
| per_mille_symbol || ë
|-
| na_n_symbol || Non un numero reale
|}

== Other information ==

=== References ===

* ms-help://MS.NETFramework.v20.en/cpref8/html/T_System_Globalization_CultureInfo.htm (ms-help is the .NET documentation)
* http://msdn2.microsoft.com/en-us/library/system.globalization.cultureinfo.aspx
* http://msdn2.microsoft.com/en-us/library/system.globalization.compareinfo.aspx

Internationalization/posix locale

2006-09-05T11:18:23Z

Etienner: /* Checking for default locale */ another method

[[Category:Internationalization]]

==Summary==
My Summary is based on article by Michael Becker (see [http://www.ijon.de/comp/tutorials/locale.html here]).This summary is more or less concerning SuSE-Linux.

There are differences between User-locale and C-locale. C-locale, is the local, which is only effective in C-environment. If there are no locales set in user environment, then C-locale is effective, which is so called "POSIX"-locale.

With C-command <code>setlocale (LC_ALL,"")</code> one set C-locale with User-locale. So one could retrieve details about User-locale information with C-functions.

==Listing available locales==
There are two directories that one could get information about locales.
They are:
''/usr/share/locale/''
''/usr/share/i18n/''

In the first directory, the locales are binary files. So we could not need them for our purpose since i do not know how to read them for the moment.

In the second directory there are at least 2 subdirectories, with names charmaps and locales.

Shell command ''ls /usr/share/i18n/locales/'' list all the available locales.

==Checking for default locale==

With shell command "locale" one can get infomation about default locale.
In C, to get details of localeone can use
<code>setlocale (LC_ALL,"")
localeconv()</code>
see ''manpage locale(7)'' to get more information.

Another way to obtain the current locale is to use the environnement variable ''LANGUAGE'', this is done with the following code:

<code>[eiffel,N]
environement : EXECUTION_ENVIRONMENT
...
environement.get ("LANGUAGE")
-- returns the (rank)list (as string) of languages set by the user
</code>

== Retrieving locale information ==
C-command <code> nl_langinfo() </code>(defined in langinfo.h) one could get information about all categories.
For example, with <code> nl_langinfo(D_T_FMT)</code>, one get a pointer at a string, which in form of %a %d %b %Y %T %Z.

== Other information ==
Using a parser one could also retrieve locale information, about which I do not have a lot of idea.

Still there are C-functions to use categories.
For example,
<code>strftime()
strptime()
wcsftime()</code>
for LC_TIME
For another example,
<code>isalnum
isalpha
isblank</code>
and so on for LC_CTYPE.

For more information please read [http://www.opengroup.org/onlinepubs/007908799/xbd/locale.html local-Specification of Unix Open]

Internationalization/dotnet locale

2006-09-05T11:16:27Z

Etienner: /* Checking for default locale */

Internationalization/dotnet locale

2006-09-04T13:18:23Z

Etienner: /* Checking for default locale */

User talk:Manus

2006-09-03T07:28:00Z

Etienner: block proposal

User talk:Manus

2006-09-03T07:24:59Z

Etienner: Vandalism...

Main Page

2006-09-03T07:23:15Z

Etienner: remove vandalism

Welcome to [http://www.eiffel.com/ Eiffel Software]'s Wiki, the home page for Eiffel Software's Open Source initiative in collaboration with the [http://se.inf.ethz.ch/ Chair of Software Engineering] at [http://www.ethz.ch ETH Zurich].

{{block|'''Project ideas''' 

If you'd like to contribute to the progress of Eiffel technology but don't know where to start, be sure to
take a look at the [[:Category:Projects| project suggestion pages]]. They contain lots of great ideas for projects large and small!}}

To download or look at the source code of EiffelStudio, please go to the [[Repository| SVN repository]] page.

To compile the source code, you will need a snapshot of the current development release of EiffelStudio. To get it go to the [[Downloads| Downloads]] page.

=== Wiki Contents ===

* [[:Category:EiffelStudio|EiffelStudio]]
* [[:Category:EiffelBuild|EiffelBuild]]
* [[:Category:Compiler| Eiffel Compiler]]
* [[:Category:Runtime| Eiffel Runtime]]
* [[:Category:Library| Eiffel Libraries]]:
** [[:Category:EiffelBase| EiffelBase]]
** [[:Category:EiffelVision2| EiffelVision2]]
** [[:Category:WEL| WEL]]
* [[:Category:Eiffel for .NET| Eiffel for .NET]]
* [[:Category:Documentation| Documentation]]
* [[Repository| SVN repository]]
* [[Contributing| How To Contribute?]]
* [[:Category:Projects| Projects and project suggestions]]
* [[URLs| Useful URLs]]
* [[TODOs| Project TODOs]]
* [[Tools| Tools]]

Internationalization/file format

2006-09-02T15:45:39Z

Etienner: /* Format of PO files */

[[Category:Internationalization]]

==Summary==

A bref description of the most importat file formats used for the translation of programs.

==PO Files==

===Format of PO files===

A PO file has an entry for each string that has to be translated. There are two kind of them, a "normal" one and one that involves plural forms.

====Normal entry====

Here is the general structure of a "normal" entry:

white-space
# translator-comments
#. extracted-comments
#: references...
#, flag...
msgid untranslated-string
msgstr translated-string

Where the ''translator-comments'' are created and maintained exclusively by the translator, this comments have some white space immediately following the #. The other comments are created by the program that created the PO file.
''References'' are space separated lists of locations (sourcefile:linenumber) specifying where the translation unit is found in a source file.
After the special comment "#," there can be some ''flags'', as ''fuzzy'' shows that the msgstr string might not be a correct translation, i.e. the translator is not sure of his work.
The 'untranslated-string' is the untranslated string as it appears in the original program source. The ''translated-string'' is (as the name suggests) the translated string, if there is no translation it is an empty string.

====Plural form entry====

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

===Supported character encodings===

character encodings that can be used are limited to those supported by both GNU libc and GNU libiconv. These are: ASCII, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-13, ISO-8859-15, KOI8-R, KOI8-U, CP850, CP866, CP874, CP932, CP949, CP950, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, GB2312, EUC-JP, EUC-KR, EUC-TW, BIG5, BIG5-HKSCS, GBK, GB18030, SHIFT_JIS, JOHAB, TIS-620, VISCII, UTF-8.

===Po Editors===

* poEdit
* KBabel
* Gtranslator
* LocFactoryEditor (XLIFF and PO editor for Mac OSX)

===Positive aspects===

* Powerful plural handling
* Format created for translation purpose
* Easy for humans to read
* Used by gettext, kbabel, rosetta and many other programs
* Support and elaboration tools for almost all plattforms

===Negative aspects===

* The PO file format does not provide a way of identifying the source and target language within a file. By GNU standards, GNU software is written in American English (en-US), and this is reflected in Gettext by only having support for Germanic plural forms in the source language. It is therefore recommended to set the source-language attribute to en-US by default.

==ts Files==

===Format of ts files===

The .ts file format is used Trolltech for the QT applications. They are XML conforming files. Here an example of a .ts file, generated my lupdate (a tool made by Trolltech that extracts translatable text from the C++ source code of the Qt application, see [[Internationalization/tool evaluation|here]] for further information):

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation type="unfinished"></translation>
</message>
</context>
</TS>

And after the translation (for example with Qt Linguist) it would look like this:

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation>i20e=Internazionalizzazione</translation>
</message>
</context>
</TS>

The .ts file is than converted to the .qm file format, a compact binary format that provides extremely fast lookups for translations, with a tool named lrelease.

The creation of .qm files can also be done with the GNU gettext tools: with "xgettext --qt" as string extractor for producing the .pot file. And then convert the translated file (.po) with the "msgfmt --qt" command for creating the .qm files.

=== ts Editors ===

*QT Linguistic

===Positive aspects===

* "full" support for unicode character encodings
* In trolltech's opinion it's a human readable text

===Negative aspects===

* QT's translation framework does not support plurals
* Qt message catalog format supports Unicode only in the translated strings, not in the untranslated strings

== xliff Files ==

===Format of XLIFF files===

XLIFF is the XML Localization Interchange File Format. It is intended to give any software provider a single interchange file format that can be understood by any localization provider.

You can find a XLIFF Tree Structure [http://www.oasis-open.org/committees/xliff/documents/xliff-specification.htm#AppTree here]

=== ts Editors ===

*[http://www.heartsome.net/EN/xlfedit.html heartsome]
*[https://open-language-tools.dev.java.net/editor/about-xliff-editor.html XLIFF Translation Editor]

===Positive aspects===

* OASIS standard

===Negative aspects===

* complicated plural form handling

==References==

* [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html Gettext manual (for PO files)]
* [http://librarian.launchpad.net/2395419/it.po Example of a PO file (from rosetta)]
* [http://news.com.com/2100-1013_3-5146581.html Microsoft and XML]
* [http://en.wikipedia.org/wiki/XML#Quick_syntax_tour XML syntax description on Wikipedia]
* [http://doc.trolltech.com/4.1/i18n.html Trolltech i18n]
* [http://translate.sourceforge.net/wiki/ open source i18n and l10n project]

Internationalization/file format

2006-09-02T15:44:53Z

Etienner: xliff...

[[Category:Internationalization]]

==Summary==

A bref description of the most importat file formats used for the translation of programs.

==PO Files==

===Format of PO files===

A PO file has an entry for each string that has to be translated. There are two kind of them, a "normal" one and one that involves plural forms.

====Normal entry====

Here is the general structure of a "normal" entry:

white-space
# translator-comments
#. extracted-comments
#: references...
#, flag...
msgid untranslated-string
msgstr translated-string

Where the ''translator-comments'' are created and maintained exclusively by the translator, this comments have some white space immediately following the #. The other comments are created by the program that created the PO file.
''References'' are space separated lists of locations (sourcefile:linenumber) specifying where the translation unit is found in a source file.
After the special comment "#," there can be some ''flags'', as ''fuzzy'' shows that the msgstr string might not be a correct translation, i.e. the translator is not sure of his work.
The 'untranslated-string' is the untranslated string as it appears in the original program source. The ''translated-string'' is (as the name suggests) the translated string, if there is no translation it is an empty string.

====Plural form entry====

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

===Supported character encodings===

character encodings that can be used are limited to those supported by both GNU libc and GNU libiconv. These are: ASCII, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-13, ISO-8859-15, KOI8-R, KOI8-U, CP850, CP866, CP874, CP932, CP949, CP950, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, GB2312, EUC-JP, EUC-KR, EUC-TW, BIG5, BIG5-HKSCS, GBK, GB18030, SHIFT_JIS, JOHAB, TIS-620, VISCII, UTF-8.

===Po Editors===

* poEdit
* KBabel
* Gtranslator
* LocFactoryEditor (XLIFF and PO editor for Mac OSX)

===Positive aspects===

* Powerful plural handling
* Format created for translation purpose
* Easy for humans to read
* Used by gettext, kbabel, rosetta and many other programs
* Support and elaboration tools for almost all plattforms

===Negative aspects===

* The PO file format does not provide a way of identifying the source and target language within a file. By GNU standards, GNU software is written in American English (en-US), and this is reflected in Gettext by only having support for Germanic plural forms in the source language. It is therefore recommended to set the source-language attribute to en-US by default.

==ts Files==

===Format of ts files===

The .ts file format is used Trolltech for the QT applications. They are XML conforming files. Here an example of a .ts file, generated my lupdate (a tool made by Trolltech that extracts translatable text from the C++ source code of the Qt application, see [[Internationalization/tool evaluation|here]] for further information):

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation type="unfinished"></translation>
</message>
</context>
</TS>

And after the translation (for example with Qt Linguist) it would look like this:

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation>i20e=Internazionalizzazione</translation>
</message>
</context>
</TS>

The .ts file is than converted to the .qm file format, a compact binary format that provides extremely fast lookups for translations, with a tool named lrelease.

The creation of .qm files can also be done with the GNU gettext tools: with "xgettext --qt" as string extractor for producing the .pot file. And then convert the translated file (.po) with the "msgfmt --qt" command for creating the .qm files.

=== ts Editors ===

*QT Linguistic

===Positive aspects===

* "full" support for unicode character encodings
* In trolltech's opinion it's a human readable text

===Negative aspects===

* QT's translation framework does not support plurals
* Qt message catalog format supports Unicode only in the translated strings, not in the untranslated strings

== xliff Files ==

===Format of PO files===

XLIFF is the XML Localization Interchange File Format. It is intended to give any software provider a single interchange file format that can be understood by any localization provider.

You can find a XLIFF Tree Structure [http://www.oasis-open.org/committees/xliff/documents/xliff-specification.htm#AppTree here]

=== ts Editors ===

*[http://www.heartsome.net/EN/xlfedit.html heartsome]
*[https://open-language-tools.dev.java.net/editor/about-xliff-editor.html XLIFF Translation Editor]

===Positive aspects===

* OASIS standard

===Negative aspects===

* complicated plural form handling

==References==

* [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html Gettext manual (for PO files)]
* [http://librarian.launchpad.net/2395419/it.po Example of a PO file (from rosetta)]
* [http://news.com.com/2100-1013_3-5146581.html Microsoft and XML]
* [http://en.wikipedia.org/wiki/XML#Quick_syntax_tour XML syntax description on Wikipedia]
* [http://doc.trolltech.com/4.1/i18n.html Trolltech i18n]
* [http://translate.sourceforge.net/wiki/ open source i18n and l10n project]

Internationalization/file format

2006-09-02T14:56:02Z

Etienner: /* Negative aspects */

[[Category:Internationalization]]

==Summary==

Here [[Internationalization|we]] evaluate various file formats used for the translation of programs. For the moment we are considering:

* XML
* po
* [http://transolution.python-hosting.com xliff] (good description in this homepage)
* create an own format

==PO Files==

===Format of PO files===

A PO file has an entry for each string that has to be translated. There are two kind of them, a "normal" one and one that involves plural forms.

====Normal entry====

Here is the general structure of a "normal" entry:

white-space
# translator-comments
#. extracted-comments
#: references...
#, flag...
msgid untranslated-string
msgstr translated-string

Where the ''translator-comments'' are created and maintained exclusively by the translator, this comments have some white space immediately following the #. The other comments are created by the program that created the PO file.
''References'' are space separated lists of locations (sourcefile:linenumber) specifying where the translation unit is found in a source file.
After the special comment "#," there can be some ''flags'', as ''fuzzy'' shows that the msgstr string might not be a correct translation, i.e. the translator is not sure of his work.
The 'untranslated-string' is the untranslated string as it appears in the original program source. The ''translated-string'' is (as the name suggests) the translated string, if there is no translation it is an empty string.

====Plural form entry====

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

===Supported character encodings===

character encodings that can be used are limited to those supported by both GNU libc and GNU libiconv. These are: ASCII, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-13, ISO-8859-15, KOI8-R, KOI8-U, CP850, CP866, CP874, CP932, CP949, CP950, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, GB2312, EUC-JP, EUC-KR, EUC-TW, BIG5, BIG5-HKSCS, GBK, GB18030, SHIFT_JIS, JOHAB, TIS-620, VISCII, UTF-8.

I think they are a lot...

===Po Editors===

* poEdit
* KBabel
* Gtranslator
* LocFactoryEditor (XLIFF and PO editor for Mac OSX)

===Positive aspects===

* Powerful plural handling
* Format created for translation purpose
* Easy for humans to read
* Used by gettext, kbabel, rosetta and many other programs
* Support and elaboration tools for almost all plattforms

===Negative aspects===

* The PO file format does not provide a way of identifying the source and target language within a file. By GNU standards, GNU software is written in American English (en-US), and this is reflected in Gettext by only having support for Germanic plural forms in the source language. It is therefore recommended to set the source-language attribute to en-US by default.

==ts Files==

===Format of ts files===

The .ts file format is used Trolltech for the QT applications. They are XML conforming files. Here an example of a .ts file, generated my lupdate (a tool made by Trolltech that extracts translatable text from the C++ source code of the Qt application, see [[Internationalization/tool evaluation|here]] for further information):

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation type="unfinished"></translation>
</message>
</context>
</TS>

And after the translation (for example with Qt Linguist) it would look like this:

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation>i20e=Internazionalizzazione</translation>
</message>
</context>
</TS>

The .ts file is than converted to the .qm file format, a compact binary format that provides extremely fast lookups for translations, with a tool named lrelease.

The creation of .qm files can also be done with the GNU gettext tools: with "xgettext --qt" as string extractor for producing the .pot file. And then convert the translated file (.po) with the "msgfmt --qt" command for creating the .qm files.

===Positive aspects===

* full support for unicode character encodings
* In trolltech's opinion it's a human readable text

===Negative aspects===

* QT's translation framework does not support plurals
* Qt message catalog format supports Unicode only in the translated strings, not in the untranslated strings

==New Format==

===Format of our Format===

* It doesn't exist yet, so we don't know how it looks like.
* We could give our own extension to the file format for example .et (eiffel translation) or .babe (babylon eiffel) or .eint (eiffel i18n) ... (''huge'' advantage)

===Positive aspects===

* Free to do what we want
* We don't have to care about licenses
* Possibility to make it the best human readable format
* So that we can say that we invented a new file format
* Better integrated and consistent with eiffel syntax

===Negative aspects===

* A new format? Why should we be different?
* Do more work as needed (there are already good formats)
* Long time until it becomes famous

==Conclusions==

Our decision is to use the po file format.

==References==

* [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html Gettext manual (for PO files)]
* [http://librarian.launchpad.net/2395419/it.po Example of a PO file (from rosetta)]
* [http://news.com.com/2100-1013_3-5146581.html Microsoft and XML]
* [http://en.wikipedia.org/wiki/XML#Quick_syntax_tour XML syntax description on Wikipedia]
* [http://doc.trolltech.com/4.1/i18n.html Trolltech i18n]
* [http://translate.sourceforge.net/wiki/ open source i18n and l10n project]

Internationalization/file format

2006-09-02T14:54:59Z

Etienner: /* Normal entry */

[[Category:Internationalization]]

==Summary==

Here [[Internationalization|we]] evaluate various file formats used for the translation of programs. For the moment we are considering:

* XML
* po
* [http://transolution.python-hosting.com xliff] (good description in this homepage)
* create an own format

==PO Files==

===Format of PO files===

A PO file has an entry for each string that has to be translated. There are two kind of them, a "normal" one and one that involves plural forms.

====Normal entry====

Here is the general structure of a "normal" entry:

white-space
# translator-comments
#. extracted-comments
#: references...
#, flag...
msgid untranslated-string
msgstr translated-string

Where the ''translator-comments'' are created and maintained exclusively by the translator, this comments have some white space immediately following the #. The other comments are created by the program that created the PO file.
''References'' are space separated lists of locations (sourcefile:linenumber) specifying where the translation unit is found in a source file.
After the special comment "#," there can be some ''flags'', as ''fuzzy'' shows that the msgstr string might not be a correct translation, i.e. the translator is not sure of his work.
The 'untranslated-string' is the untranslated string as it appears in the original program source. The ''translated-string'' is (as the name suggests) the translated string, if there is no translation it is an empty string.

====Plural form entry====

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

===Supported character encodings===

character encodings that can be used are limited to those supported by both GNU libc and GNU libiconv. These are: ASCII, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-13, ISO-8859-15, KOI8-R, KOI8-U, CP850, CP866, CP874, CP932, CP949, CP950, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, GB2312, EUC-JP, EUC-KR, EUC-TW, BIG5, BIG5-HKSCS, GBK, GB18030, SHIFT_JIS, JOHAB, TIS-620, VISCII, UTF-8.

I think they are a lot...

===Po Editors===

* poEdit
* KBabel
* Gtranslator
* LocFactoryEditor (XLIFF and PO editor for Mac OSX)

===Positive aspects===

* Powerful plural handling
* Format created for translation purpose
* Easy for humans to read
* Used by gettext, kbabel, rosetta and many other programs
* Support and elaboration tools for almost all plattforms

===Negative aspects===

* We have to write a PO parser

==ts Files==

===Format of ts files===

The .ts file format is used Trolltech for the QT applications. They are XML conforming files. Here an example of a .ts file, generated my lupdate (a tool made by Trolltech that extracts translatable text from the C++ source code of the Qt application, see [[Internationalization/tool evaluation|here]] for further information):

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation type="unfinished"></translation>
</message>
</context>
</TS>

And after the translation (for example with Qt Linguist) it would look like this:

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation>i20e=Internazionalizzazione</translation>
</message>
</context>
</TS>

The .ts file is than converted to the .qm file format, a compact binary format that provides extremely fast lookups for translations, with a tool named lrelease.

The creation of .qm files can also be done with the GNU gettext tools: with "xgettext --qt" as string extractor for producing the .pot file. And then convert the translated file (.po) with the "msgfmt --qt" command for creating the .qm files.

===Positive aspects===

* full support for unicode character encodings
* In trolltech's opinion it's a human readable text

===Negative aspects===

* QT's translation framework does not support plurals
* Qt message catalog format supports Unicode only in the translated strings, not in the untranslated strings

==New Format==

===Format of our Format===

* It doesn't exist yet, so we don't know how it looks like.
* We could give our own extension to the file format for example .et (eiffel translation) or .babe (babylon eiffel) or .eint (eiffel i18n) ... (''huge'' advantage)

===Positive aspects===

* Free to do what we want
* We don't have to care about licenses
* Possibility to make it the best human readable format
* So that we can say that we invented a new file format
* Better integrated and consistent with eiffel syntax

===Negative aspects===

* A new format? Why should we be different?
* Do more work as needed (there are already good formats)
* Long time until it becomes famous

==Conclusions==

Our decision is to use the po file format.

==References==

* [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html Gettext manual (for PO files)]
* [http://librarian.launchpad.net/2395419/it.po Example of a PO file (from rosetta)]
* [http://news.com.com/2100-1013_3-5146581.html Microsoft and XML]
* [http://en.wikipedia.org/wiki/XML#Quick_syntax_tour XML syntax description on Wikipedia]
* [http://doc.trolltech.com/4.1/i18n.html Trolltech i18n]
* [http://translate.sourceforge.net/wiki/ open source i18n and l10n project]

Internationalization/file format

2006-09-02T09:01:47Z

Etienner: /* ts Files */ -> Negative aspects

[[Category:Internationalization]]

==Summary==

Here [[Internationalization|we]] evaluate various file formats used for the translation of programs. For the moment we are considering:

* XML
* po
* [http://transolution.python-hosting.com xliff] (good description in this homepage)
* create an own format

==PO Files==

===Format of PO files===

A PO file has an entry for each string that has to be translated. There are two kind of them, a "normal" one and one that involves plural forms.

====Normal entry====

Here is the general structure of a "normal" entry:

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string
msgstr translated-string

Where the ''translator-comments'' are created and maintained exclusively by the translator, this comments have some white space immediately following the #. The other comments are created by the program that created the PO file.
After the special comment "#," there can be some ''flags'', as ''fuzzy'' shows that the msgstr string might not be a correct translation, i.e. the translator is not sure of his work.
The 'untranslated-string' is the untranslated string as it appears in the original program source. The ''translated-string'' is (as the name suggests) the translated string, if there is no translation it is an empty string.

====Plural form entry====

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

===Supported character encodings===

character encodings that can be used are limited to those supported by both GNU libc and GNU libiconv. These are: ASCII, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-13, ISO-8859-15, KOI8-R, KOI8-U, CP850, CP866, CP874, CP932, CP949, CP950, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, GB2312, EUC-JP, EUC-KR, EUC-TW, BIG5, BIG5-HKSCS, GBK, GB18030, SHIFT_JIS, JOHAB, TIS-620, VISCII, UTF-8.

I think they are a lot...

===Po Editors===

* poEdit
* KBabel
* Gtranslator
* LocFactoryEditor (XLIFF and PO editor for Mac OSX)

===Positive aspects===

* Powerful plural handling
* Format created for translation purpose
* Easy for humans to read
* Used by gettext, kbabel, rosetta and many other programs
* Support and elaboration tools for almost all plattforms

===Negative aspects===

* We have to write a PO parser

==ts Files==

===Format of ts files===

The .ts file format is used Trolltech for the QT applications. They are XML conforming files. Here an example of a .ts file, generated my lupdate (a tool made by Trolltech that extracts translatable text from the C++ source code of the Qt application, see [[Internationalization/tool evaluation|here]] for further information):

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation type="unfinished"></translation>
</message>
</context>
</TS>

And after the translation (for example with Qt Linguist) it would look like this:

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation>i20e=Internazionalizzazione</translation>
</message>
</context>
</TS>

The .ts file is than converted to the .qm file format, a compact binary format that provides extremely fast lookups for translations, with a tool named lrelease.

The creation of .qm files can also be done with the GNU gettext tools: with "xgettext --qt" as string extractor for producing the .pot file. And then convert the translated file (.po) with the "msgfmt --qt" command for creating the .qm files.

===Positive aspects===

* full support for unicode character encodings
* In trolltech's opinion it's a human readable text

===Negative aspects===

* QT's translation framework does not support plurals
* Qt message catalog format supports Unicode only in the translated strings, not in the untranslated strings

==New Format==

===Format of our Format===

* It doesn't exist yet, so we don't know how it looks like.
* We could give our own extension to the file format for example .et (eiffel translation) or .babe (babylon eiffel) or .eint (eiffel i18n) ... (''huge'' advantage)

===Positive aspects===

* Free to do what we want
* We don't have to care about licenses
* Possibility to make it the best human readable format
* So that we can say that we invented a new file format
* Better integrated and consistent with eiffel syntax

===Negative aspects===

* A new format? Why should we be different?
* Do more work as needed (there are already good formats)
* Long time until it becomes famous

==Conclusions==

Our decision is to use the po file format.

==References==

* [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html Gettext manual (for PO files)]
* [http://librarian.launchpad.net/2395419/it.po Example of a PO file (from rosetta)]
* [http://news.com.com/2100-1013_3-5146581.html Microsoft and XML]
* [http://en.wikipedia.org/wiki/XML#Quick_syntax_tour XML syntax description on Wikipedia]
* [http://doc.trolltech.com/4.1/i18n.html Trolltech i18n]
* [http://translate.sourceforge.net/wiki/ open source i18n and l10n project]

Internationalization/file format

2006-09-02T08:39:22Z

Etienner: /* XML */

[[Category:Internationalization]]

==Summary==

Here [[Internationalization|we]] evaluate various file formats used for the translation of programs. For the moment we are considering:

* XML
* po
* [http://transolution.python-hosting.com xliff] (good description in this homepage)
* create an own format

==PO Files==

===Format of PO files===

A PO file has an entry for each string that has to be translated. There are two kind of them, a "normal" one and one that involves plural forms.

====Normal entry====

Here is the general structure of a "normal" entry:

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string
msgstr translated-string

Where the ''translator-comments'' are created and maintained exclusively by the translator, this comments have some white space immediately following the #. The other comments are created by the program that created the PO file.
After the special comment "#," there can be some ''flags'', as ''fuzzy'' shows that the msgstr string might not be a correct translation, i.e. the translator is not sure of his work.
The 'untranslated-string' is the untranslated string as it appears in the original program source. The ''translated-string'' is (as the name suggests) the translated string, if there is no translation it is an empty string.

====Plural form entry====

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

===Supported character encodings===

character encodings that can be used are limited to those supported by both GNU libc and GNU libiconv. These are: ASCII, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-13, ISO-8859-15, KOI8-R, KOI8-U, CP850, CP866, CP874, CP932, CP949, CP950, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, GB2312, EUC-JP, EUC-KR, EUC-TW, BIG5, BIG5-HKSCS, GBK, GB18030, SHIFT_JIS, JOHAB, TIS-620, VISCII, UTF-8.

I think they are a lot...

===Po Editors===

* poEdit
* KBabel
* Gtranslator
* LocFactoryEditor (XLIFF and PO editor for Mac OSX)

===Positive aspects===

* Powerful plural handling
* Format created for translation purpose
* Easy for humans to read
* Used by gettext, kbabel, rosetta and many other programs
* Support and elaboration tools for almost all plattforms

===Negative aspects===

* We have to write a PO parser

==ts Files==

===Format of ts files===

The .ts file format is used Trolltech for the QT applications. They are XML conforming files. Here an example of a .ts file, generated my lupdate (a tool made by Trolltech that extracts translatable text from the C++ source code of the Qt application, see [[Internationalization/tool evaluation|here]] for further information):

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation type="unfinished"></translation>
</message>
</context>
</TS>

And after the translation (for example with Qt Linguist) it would look like this:

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation>i20e=Internazionalizzazione</translation>
</message>
</context>
</TS>

The .ts file is than converted to the .qm file format, a compact binary format that provides extremely fast lookups for translations, with a tool named lrelease.

The creation of .qm files can also be done with the GNU gettext tools: with "xgettext --qt" as string extractor for producing the .pot file. And then convert the translated file (.po) with the "msgfmt --qt" command for creating the .qm files.

===Positive aspects===

* full support for unicode character encodings
* In trolltech's opinion it's a human readable text

===Negative aspects===

* Not everybody knows it
* QT's translation framework does not support plurals

==New Format==

===Format of our Format===

* It doesn't exist yet, so we don't know how it looks like.
* We could give our own extension to the file format for example .et (eiffel translation) or .babe (babylon eiffel) or .eint (eiffel i18n) ... (''huge'' advantage)

===Positive aspects===

* Free to do what we want
* We don't have to care about licenses
* Possibility to make it the best human readable format
* So that we can say that we invented a new file format
* Better integrated and consistent with eiffel syntax

===Negative aspects===

* A new format? Why should we be different?
* Do more work as needed (there are already good formats)
* Long time until it becomes famous

==Conclusions==

Our decision is to use the po file format.

==References==

* [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html Gettext manual (for PO files)]
* [http://librarian.launchpad.net/2395419/it.po Example of a PO file (from rosetta)]
* [http://news.com.com/2100-1013_3-5146581.html Microsoft and XML]
* [http://en.wikipedia.org/wiki/XML#Quick_syntax_tour XML syntax description on Wikipedia]
* [http://doc.trolltech.com/4.1/i18n.html Trolltech i18n]
* [http://translate.sourceforge.net/wiki/ open source i18n and l10n project]

Internationalization/dotnet locale

2006-09-01T15:19:11Z

Etienner: Checking for default locale

Internationalization/dotnet locale

2006-09-01T15:03:48Z

Etienner: /* Links */

Internationalization/dotnet locale

2006-09-01T14:54:13Z

Etienner: Style

Internationalization/obstacles

2006-06-29T17:08:49Z

Etienner: small syntax fix

Problems we encountered during implementation

* Unicode (UTF-8) support was not fully implemented (e.g. for gtk)
* Facilities to write and read STRING_32 to/from files were not present

Difficulties to complete the translation of EiffelStudio:
* Many strings are hard-coded anywhere inside EiffelStudio, making the translation difficult
* All the functions taking strings as arguments expect a STRING, where they should expect a STRING_GENERAL, and they return a STRING which may not be applicable in every case. A STRING_32 can also represent the same content of a STRING_8, thus it should be preferred.
* It seems that to_uppercase and to_lowercase procedures are not defined for STRING_32 and are used in the studio, these should have been implemented by the vision project and may be buggy.

User:Etienner

2006-06-28T18:26:03Z

Etienner:

==Userpage of Etienne Reichenbach==

I am involved in the following projects:

* [[Internationalization]]
** [http://eiffelsoftware.origo.ethz.ch/index.php/Category:Internationalization Article list]

Internationalization

2006-06-28T18:23:14Z

Etienner: /* M5: July ?? */

[[Category:Projects]]
[[Category:Internationalization]]
[[Image:ebabylon.png|right|frame| Our Eiffel Tower of Babylon]]

=Overview=

''"Many [people] would simply love seeing their computer screen showing a lot less of English, and far more of their own language."'' -- gettext doc

Our aim is not only to provide a framework to ease the translation of Eiffel-written applications, allowing the user to chose his/her preferred language at runtime, but also to let the developer access information and formats based on users' locale.

==What is internationalisation?==

The first thing that comes to mind is translation. But internationalisation isn't restricted to enabling translation: it includes making it possible to localise notations (time, date, numbers), measures, paper size, and much more.

==What should we achieve?==
*Applications should be able to load localized strings at runtime and be provided with localized format strings (e.g date format).
*Developers can use tools that automagically extract strings from source code and can try to get them translated in a file to distribute along with the application.
*Users will still be unhappy and get depressed ''but in their own language'', which we can all agree is a significant step forward.

=Milestones=

==M2: May 5 ==
* [[Internationalization/feasibility|feasibility]]: look at string classes (unicode and not) and how strings are used in EiffelStudio (creation, composition) ''(Ivano, Carlo, Leo, Hong)''
* [[Internationalization/file_format|file format]]: compare existing file formats for dictionaries ''(Etienne, Andreas)''
* [[Internationalization/tool_evaluation|tool evaluation]]: list and compare existing translation tools ''(Christian, Martino)''

==M3: June 13 ==
* write an initial .po to start translating and have a .mo for testing ''(Leo, Carlo)''
* [[Internationalization/mo parser|mo parser]]: extract translated strings from .mo files ''([[User:etienner|Etienne]], [[User:Trosim|Martino]])''
* [[Internationalization/translation function|translation function]]: map hard-coded strings to translated strings ''(Martino)''
** find a solution with templates
** globality: how to implement, the object should be shared between all modules (see [[Internationalization/class_structure|class structure]])
** find out how to use plurals (see [[Internationalization/plural_forms|plural forms]])
* Test unicode support in Vision2. [See test application in the [https://eiffelsoftware.origo.ethz.ch/svn/es/branches/soft-arch/Src/library/i18n/example/ SVN repository]]
* compile a [[Internationalization/features list|list of basic features]] to provide (e.g. date/time format, system locale) ''[deferred]''

==M4: June 28 ==
* internationalization of EiffelStudio: surround strings with our functions
* [[Internationalization/translation|translation]] of EiffelStudio in some languages, for demo purposes (Italian, German, Chinese, ...) (see [http://slhk.ath.cx/projects/estudio/ pootle])
* [[Internationalization/code_parser|code parser]]: extract strings to be translated from source code and generate .pot file ''(Leo)''
* language selection: add an entry in the preferences system
* create a function to detect current environment language and settings (aka [[Internationalization/locale|locale]])

==M5: July ?? ==
* Refactoring of whole library
* Use our library for EiffelStudio
* Clean up Wiki and improve Developer manual

= Possible future developments =

* collaborate with the [[ESWizard]] team. Create wizards for programs with translation facilities.

= Documentation =

* [[Internationalization/requirements_specification|Requirements specification]]
* [[Internationalization/developer_manual|Developer manual]]
* [[Internationalization/obstacles|Obstacles]]: why can't achieve all of our goals

= l10n: completed and on the work =

{| border="0" cellspacing="3" cellpadding="5" align="center"
!'''Language'''
!'''Status'''
|-
|Arabic
|Active
|-
|Chinese
|Active
|-
|Italian
|Active
|-
|Rhaeto-Romanic
|Active - Ready for use
|}

=Relevant Links=
[http://www.debian.org/doc/manuals/intro-i18n/ Introduction to i18n]

==What other people have done==
[http://doc.trolltech.com/4.1/i18n.html internationalisation with QT]

[http://oss.erdfunkstelle.de/kde-i18n/tiki-index.php?page=miniHowtoGui howto for internationalisation of KDE programs ]

[http://l10n.kde.org/docs/translation-howto/ another KDE howto (doesn't Gnome do any internationalisation?)]

=Team=

Everyone interested in this project is welcome to [http://origo.ethz.ch/cgi-bin/mailman/listinfo/es-i18n join our mailinglist] es-i18n@origo.ethz.ch

* Project Leader: [[User:Carlo|Carlo Vanini]]
* [[User:leo| Leo Fellmann]]
* [[User:kiwi| Ivano Somaini]]
* [[User:murbi|Andreas Murbach]]
* [[User:etienner|Etienne Reichenbach]]
* [[User:hong |Hong Zhang]]
* [[User:cconti | Christian Conti]]
* [[User:Trosim |Martino Trosi]]
* [[User:Schoelle| Bernd Schoeller]]

Internationalization

2006-06-28T15:25:34Z

Etienner: /* Milestones */

[[Category:Projects]]
[[Category:Internationalization]]
[[Image:ebabylon.png|right|frame| Our Eiffel Tower of Babylon]]

=Overview=

''"Many [people] would simply love seeing their computer screen showing a lot less of English, and far more of their own language."'' -- gettext doc

Our aim is not only to provide a framework to ease the translation of Eiffel-written applications, allowing the user to chose his/her preferred language at runtime, but also to let the developer access information and formats based on users' locale.

==What is internationalisation?==

The first thing that comes to mind is translation. But internationalisation isn't restricted to enabling translation: it includes making it possible to localise notations (time, date, numbers), measures, paper size, and much more.

==What should we achieve?==
*Applications should be able to load localized strings at runtime and be provided with localized format strings (e.g date format).
*Developers can use tools that automagically extract strings from source code and can try to get them translated in a file to distribute along with the application.
*Users will still be unhappy and get depressed ''but in their own language'', which we can all agree is a significant step forward.

=Milestones=

==M2: May 5 ==
* [[Internationalization/feasibility|feasibility]]: look at string classes (unicode and not) and how strings are used in EiffelStudio (creation, composition) ''(Ivano, Carlo, Leo, Hong)''
* [[Internationalization/file_format|file format]]: compare existing file formats for dictionaries ''(Etienne, Andreas)''
* [[Internationalization/tool_evaluation|tool evaluation]]: list and compare existing translation tools ''(Christian, Martino)''

==M3: June 13 ==
* write an initial .po to start translating and have a .mo for testing ''(Leo, Carlo)''
* [[Internationalization/mo parser|mo parser]]: extract translated strings from .mo files ''([[User:etienner|Etienne]], [[User:Trosim|Martino]])''
* [[Internationalization/translation function|translation function]]: map hard-coded strings to translated strings ''(Martino)''
** find a solution with templates
** globality: how to implement, the object should be shared between all modules (see [[Internationalization/class_structure|class structure]])
** find out how to use plurals (see [[Internationalization/plural_forms|plural forms]])
* Test unicode support in Vision2. [See test application in the [https://eiffelsoftware.origo.ethz.ch/svn/es/branches/soft-arch/Src/library/i18n/example/ SVN repository]]
* compile a [[Internationalization/features list|list of basic features]] to provide (e.g. date/time format, system locale) ''[deferred]''

==M4: June 28 ==
* internationalization of EiffelStudio: surround strings with our functions
* [[Internationalization/translation|translation]] of EiffelStudio in some languages, for demo purposes (Italian, German, Chinese, ...) (see [http://slhk.ath.cx/projects/estudio/ pootle])
* [[Internationalization/code_parser|code parser]]: extract strings to be translated from source code and generate .pot file ''(Leo)''
* language selection: add an entry in the preferences system
* create a function to detect current environment language and settings (aka [[Internationalization/locale|locale]])

==M5: July ?? ==
* Refactoring of whole library
* Use our library for EiffelStudio

= Possible future developments =

* collaborate with the [[ESWizard]] team. Create wizards for programs with translation facilities.

= Documentation =

* [[Internationalization/requirements_specification|Requirements specification]]
* [[Internationalization/developer_manual|Developer manual]]
* [[Internationalization/obstacles|Obstacles]]: why can't achieve all of our goals

= l10n: completed and on the work =

{| border="0" cellspacing="3" cellpadding="5" align="center"
!'''Language'''
!'''Status'''
|-
|Arabic
|Active
|-
|Chinese
|Active
|-
|Italian
|Active
|-
|Rhaeto-Romanic
|Active - Ready for use
|}

=Relevant Links=
[http://www.debian.org/doc/manuals/intro-i18n/ Introduction to i18n]

==What other people have done==
[http://doc.trolltech.com/4.1/i18n.html internationalisation with QT]

[http://oss.erdfunkstelle.de/kde-i18n/tiki-index.php?page=miniHowtoGui howto for internationalisation of KDE programs ]

[http://l10n.kde.org/docs/translation-howto/ another KDE howto (doesn't Gnome do any internationalisation?)]

=Team=

Everyone interested in this project is welcome to [http://origo.ethz.ch/cgi-bin/mailman/listinfo/es-i18n join our mailinglist] es-i18n@origo.ethz.ch

* Project Leader: [[User:Carlo|Carlo Vanini]]
* [[User:leo| Leo Fellmann]]
* [[User:kiwi| Ivano Somaini]]
* [[User:murbi|Andreas Murbach]]
* [[User:etienner|Etienne Reichenbach]]
* [[User:hong |Hong Zhang]]
* [[User:cconti | Christian Conti]]
* [[User:Trosim |Martino Trosi]]
* [[User:Schoelle| Bernd Schoeller]]

Internationalization/developer manual

2006-06-26T05:53:40Z

Etienner: /* Translating with templates */

[[Category:Internationalization]]

= How to design the software =
There is only a few things which can help you using our library, the most important one is dividing completely the logic of the program from the content.
In other words, you should write a class which contains all the strings you use in the program; so applying out system will be very easy.

= How to use the library =

Here you can find instructions on how to use our library; how to initialize the system and how to make the translations actually appear in your application.

== Initialization ==

To initialize the library you should have a class that inherits from the SHARED_I18N_LOCALIZATOR; this will bring you all the necessary infrastructure to start localizing your software.

What to do?
* create datasource
* create datastructure
* load localizator

=== Creating a datasource ===

We equip you with a simple factory to create the sources, simply create an I18N_DATASOURCE_FACTORY

<code>[eiffel,N]
datasource_factory: I18N_DATASOURCE_FACTORY

create datasource_factory.make
</code>

you now have the possibility to create a new source based on an mo file, as follows

<code>[eiffel,N]
datasource: I18N_DATASOURCE

datasource_factory.use_mo_file(mo_file_path)
if datasource_factory.last_datasource /= Void then
datasource := datasource_factory.last_datasource
end
</code>

if something went wrong, you can alway fallback with an empty datasource

<code>[eiffel,N]
datasource_factory.use_empty_source
datasource := datasource_factory.last_datasource
</code>

The datasource is ready, you should create a datastructure for storing the strings.

=== Creating a datastructure ===

We provide you with a simple factory to create datastructures, so follow the example

<code>[eiffel,N]
datastructure_factory: I18N_DATASTRUCTURE_FACTORY

create datastructure_factory.make
</code>

You can now create, for example, an hash table

<code>[eiffel,N]
datastructure: I18N_DATASTRUCTURE

datastructure_factory.use_hash_table
if datastructure_factory.last_datastructure /= Void then
datastructure := datastructure_factory.last_datastructure
end
</code>

and, even in this case, if something went wrong, you can create a dummy datastructure

<code>[eiffel,N]
datastructure_factory.use_dummy
datastructure := datastructure_factory.last_datastructure
</code>

Now that you have the datastructure ready, you can load the localizator.

=== Loading the localizator ===

It's important that you do this part before trying to localize any string, if not, the localizator would serve you what you pass as argument.

Pass the source to the localizator

<code>[eiffel,N]
i18n_use_datasource(datasource)
</code>

then pass the datastructure

<code>[eiffel,N]
i18n_use_datastructure(datastructure)
</code>

and finally load the localizator with the new source and structure

<code>[eiffel,N]
i18n_load
</code>

You now have a working localization environment!

{{Warning|If you don't assign a source AND a structure AND then call load, the system will continue to use the old ones.}}

== Localization ==

Here you can find some instructions to translate the strings of you program, including substituting variables into templates.

=== Translating simple strings ===

If you have a simple string to translate, like this

<code>[eiffel,N]
simple_string: STRING

simple_string := "A simple string"
</code>

you only have to enclose the string by the simple i18n() function

<code>[eiffel,N]
simple_string := i18n("A simple string")
</code>

the system will then be charged of the translation in whatever language you've chosen.

=== Translating plurals ===

Sometimes you must change the translated string in relation with a variable; this piece of code

<code>[eiffel,N]
string: STRING
i: INTEGER

if i = 1 then
string := "There is 1 file"
else
string := "There are more files"
end
</code>

will be written as follows using our system

<code>[eiffel,N]
string := i18n_pl("There is 1 file", "There are more files", i)
</code>

the right form will be selected and displayed for you.

=== Translating with templates ===

{{Warning|We encourage you to use a stand alone template engine and not rely on our built-in functions, which will not be maintained.}}

You can use our library to translate something like this

<code>[eiffel,N]
string := "Number " + i.out
</code>

in this way

<code>[eiffel,N]
string := i18n_comp("Number $1", [i])
</code>

and of course you can do the same with the plural forms

<code>[eiffel,N]
if i = 1 then
string := "There is 1 file"
else
string := "There are " + i.out + " files
</code>

would look like this

<code>[eiffel,N]
string := i18n_comp_pl("There is 1 file", "There are $1 files", [i], i)
</code>

we've so covered all the possibilities.

= How to translate / use translation files =

== Editing PO files ==

You can edit the PO files created with your preferred editor; some possible tools are listed [[Internationalization/tool_evaluation|here]].

== Issues with MO files ==

Into the header section of the PO file must be present the "Plural-Forms" header.

It looks like this line:

"Plural-Forms: nplurals=2; plural=(n!=1);"

with "nplural" being the number of plurals of the destination language and "plural" the C-like function for finding the right plural form starting with an integer.

Internationalization/locale

2006-06-25T06:34:50Z

Etienner: /* Linux */

[[Category:Internationalization]]

In computing, locale is a set of parameters that defines the user's language, country and any special variant preferences that the user wants to see in their user interface.

== Format of locale on OS's ==
* windows (for unmanaged code): hexadecimal code consisting of a language code (lower 10 bits) and culture code (upper bits), aka Locale Identifier (LCID)
* linux, unix: defined as [language[_territory][.codeset][@modifier]]

== Where to find locale ==
=== Linux ===
the environment variable is LANG

here is a list for more specific variables:

{| border="0" cellspacing="3" cellpadding="5" align="center"
! '''Locale category'''
! '''Application'''
|-
|LC_COLLATE
|Collation of strings (sort order.)
|-
|LC_CTYPE
|Classification and conversion of characters.
|-
|LC_MESSAGES
|Translations of yes and no.
|-
|LC_MONETARY
|Format of monetary values.
|-
|LC_NUMERIC
|Format of non-monetary numeric values.
|-
|LC_TIME
|Date and time formats.
|-
|LC_ALL
|Sets all of the above (overrides all of them.)
|-
|LANG
|Sets all the categories, but can be overridden by the individual locale categories.
|}

=== Windows ===
The LCID for "Language for non-Unicode programs" (aka system locale) can be found in the windows registry. 
The LCID of "standards and formats" (aka user locale) doesn't seem to be in the registry nor in the environment variables of windows.

== How to access to locale ==
=== Linux ===
The function "get" that fetches the language environment variable (LANG) can be found in class EXECUTION_ENVIRONMENT

SHARED_EXEC_ENVIRONMENT: located [https://eiffelsoftware.origo.ethz.ch/svn/es/branches/soft-arch/Src/Eiffel/eiffel/shared/shared_exec_environment.e here]

=== Windows ===
In Eiffel there exist a "locale_id" in class WEL_COMPARE_ITEM_STRUCT that uses the macro MAKELCID to get the locale identifier (aka LCID) from the language id.

Two solutions: 
* Find out how to get the language id
* Use the locale id (convenient if SortID is needed)

Notes: 
Locale id is formed by SortID and LanguageID. 
SortID contains info on the language encoding (?) like japanese unicode order, japanese XJIS order, chinese unicode order, chinese BIG5 order,... 
Macros for windows are contained in "windows.h".

'''Found the macro needed: GetUserDefaultLCID()'''

The code should then look like this:

<code>[eiffel,N]
language_id: NATURAL_32 is
external
"C inline use <windows.h>"
alias
"return GetUserDefaultLCID();"
end
</code>

== OS detection ==
Queries for OS detection cam be found in class PLATFORM

== References and useful links ==
* [http://www-950.ibm.com/software/globalization/icu/demo/locales info] about what's contained in a locale (language codes, time format,...)
* [http://en.wikipedia.org/wiki/Date_and_time_notation_by_country date and time notations]
* [http://en.wikipedia.org/wiki/Locale Locale on wikipedia]

Internationalization/locale

2006-06-19T10:51:10Z

Etienner: Category

[[Category:Internationalization]]

== Definition (from wikipedia: [http://en.wikipedia.org/wiki/Locale Locale])==
In computing, locale is a set of parameters that defines the user's language, country and any special variant preferences that the user wants to see in their user interface.

== Format of locale on OS's (from wikipedia: [http://en.wikipedia.org/wiki/Locale Locale]) ==
* windows (for unmanaged code): hexadecimal code consisting of a language code (lower 10 bits) and culture code (upper bits), aka Locale Identifier (LCID)
* linux, unix: defined as [language[_territory][.codeset][@modifier]]

== Useful links ==
* [http://www-950.ibm.com/software/globalization/icu/demo/locales info] about what's contained in a locale (language codes, time format,...)
* [http://en.wikipedia.org/wiki/Date_and_time_notation_by_country date and time notations]

User:Etienner

2006-06-11T16:00:13Z

Etienner:

I am involved in the following projects:

* [[Internationalization]]
:* [http://eiffelsoftware.origo.ethz.ch/index.php/Category:Internationalization Article list]

Internationalization/translation

2006-06-09T11:53:04Z

Etienner:

[[Category:Internationalization]]

The translation of EiffelStudio is going on [http://slhk.ath.cx/projects/estudio/ here].

EiffelStudio contains many sectorial terms, wich may have a corresponding expression in other languages but their meaning may not be clear enough to the end user.

This page is intended for discussion on terms difficult to translate.

----

Ancestor
* it:Antenato

Callers
* it: chiamanti

Parent

Put Handle Left

Step Out

Suppliers
* it: contribuenti

Toggle visibility
* it: Mostra/nascondi ''[used]''

Wizard
* it: Procedura guidata ''[used]'', wizard ''[used in menu]''

----
Following words are left unchanged
* it: file, inheritance, overflow, stack, feature, routine

Internationalization

2006-06-09T05:41:46Z

Etienner:

[[Category:Projects]]
[[Category:Internationalization]]
[[Image:ebabylon.png|right|frame| Our Eiffel Tower of Babylon]]

=Overview=

''"Many [people] would simply love seeing their computer screen showing a lot less of English, and far more of their own language."'' -- gettext doc

Our aim is not only to provide a framework to ease the translation of Eiffel-written applications, allowing the user to chose his/her preferred language at runtime, but also to let the developer access information and formats based on users' locale.

==What is internationalisation?==

The first thing that comes to mind is translation. But internationalisation isn't restricted to enabling translation: it includes making it possible to localise notations (time, date, numbers), measures, paper size, and much more.

==What should we achieve?==
*Applications should be able to load localized strings at runtime and be provided with localized format strings (e.g date format).
*Developers can use tools that automagically extract strings from source code and can try to get them translated in a file to distribute along with the application.
*Users will still be unhappy and get depressed ''but in their own language'', which we can all agree is a significant step forward.

=Milestones=

==M2: May 5 ==
* [[Internationalization/feasibility|feasibility]]: look at string classes (unicode and not) and how strings are used in EiffelStudio (creation, composition) ''(Ivano, Carlo, Leo, Hong)''
* [[Internationalization/file_format|file format]]: compare existing file formats for dictionaries ''(Etienne, Andreas)''
* [[Internationalization/tool_evaluation|tool evaluation]]: list and compare existing translation tools ''(Christian, Martino)''

==M3: June 13 ==
* write an initial .po to start translating and have a .mo for testing ''(Leo, Carlo)''
* [[Internationalization/mo parser|mo parser]]: extract translated strings from .mo files ''([[User:etienner|Etienne]], [[User:Trosim|Martino]])''
* [[Internationalization/translation function|translation function]]: map hard-coded strings to translated strings ''(Martino)''
** find a solution with templates
** globality: how to implement, the object should be shared between all modules (see [[Internationalization/class_structure|class structure]])
** find out how to use plurals (see [[Internationalization/plural_forms|plural forms]])
* Test unicode support in Vision2. [See test application in the [https://eiffelsoftware.origo.ethz.ch/svn/es/branches/soft-arch/Src/library/i18n/example/ SVN repository]]
* [[Internationalization/code_parser|code parser]]: extract strings to be translated from source code and generate .po file ''(Leo)'' ''[deferred]''
* compile a [[Internationalization/features list|list of basic features]] to provide (e.g. date/time format, system locale) ''[deferred]''

==M4: June ??? ==
* internationalization of EiffelStudio: surround strings with our functions
* translation of EiffelStudio in some languages, for demo purposes (Italian, German, Chinese, ...) (see [http://slhk.ath.cx/projects/estudio/ pootle])
* [[Internationalization/code_parser|code parser]]: extract strings to be translated from source code and generate .pot file ''(Leo)''
* language selection: add an entry in the preferences system

= Possible future developments =

* collaborate with the [[ESWizard]] team. Create wizards for programms with translation facilities.

=Relevant Links=
[http://www.debian.org/doc/manuals/intro-i18n/ Introduction to i18n]

==What other people have done==
[http://doc.trolltech.com/4.1/i18n.html internationalisation with QT]

[http://oss.erdfunkstelle.de/kde-i18n/tiki-index.php?page=miniHowtoGui howto for internationalisation of KDE programs ]

[http://l10n.kde.org/docs/translation-howto/ another KDE howto (doesn't Gnome do any internationalisation?)]

=Team=

Everyone interested in this project is welcome to [http://origo.ethz.ch/cgi-bin/mailman/listinfo/es-i18n join our mailinglist] es-i18n@origo.ethz.ch

* Project Leader: [[User:Carlo|Carlo Vanini]]
* [[User:leo| Leo Fellmann]]
* [[User:kiwi| Ivano Somaini]]
* [[User:murbi|Andreas Murbach]]
* [[User:etienner|Etienne Reichenbach]]
* [[User:hong |Hong Zhang]]
* [[User:cconti | Christian Conti]]
* [[User:Trosim |Martino Trosi]]
* [[User:Schoelle| Bernd Schoeller]]

Internationalization

2006-06-07T18:57:28Z

Etienner: /* M3: June ??? */

[[Category:Projects]]
[[Category:Internationalization]]
[[Image:ebabylon.png|right|frame| Our Eiffel Tower of Babylon]]

=Overview=

''"Many [people] would simply love seeing their computer screen showing a lot less of English, and far more of their own language."'' -- gettext doc

Our aim is not only to provide a framework to ease the translation of Eiffel-written applications, allowing the user to chose his/her preferred language at runtime, but also to let the developer access information and formats based on users' locale.

==What is internationalisation?==

The first thing that comes to mind is translation. But internationalisation isn't restricted to enabling translation: it includes making it possible to localise notations (time, date, numbers), measures, paper size, and much more.

==What should we achieve?==
*Applications should be able to load localized strings at runtime and be provided with localized format strings (e.g date format).
*Developers can use tools that automagically extract strings from source code and can try to get them translated in a file to distribute along with the application.
*Users will still be unhappy and get depressed ''but in their own language'', which we can all agree is a significant step forward.

==Who should do the achieving?==
It is clear that our [[User:Carlo|Glorious Leader]] should be credited with any achieving. However he will have to delegate things somewhat, as he is is not [http://en.wikipedia.org/wiki/Aleksei_Grigorievich_Stakhanov Stakhanov].
We can probably divide the project into the following parts, modulo preliminary research:

''This should be probably be completed after we have got past M1''

=Milestones=

==M2: May 5 ==
* [[Internationalization/feasibility|feasibility]]: look at string classes (unicode and not) and how strings are used in EiffelStudio (creation, composition) ''(Ivano, Carlo, Leo, Hong)''
* [[Internationalization/file_format|file format]]: compare existing file formats for dictionaries ''(Etienne, Andreas)''
* [[Internationalization/tool_evaluation|tool evaluation]]: list and compare existing translation tools ''(Christian, Martino)''

==M3: June ??? ==
* write an initial .po to start translating and have a .mo for testing ''(Leo, Carlo)''
* [[Internationalization/mo parser|mo parser]]: extract translated strings from .mo files ''([[User:etienner|Etienne]], [[User:Trosim|Martino]])''
* [[Internationalization/code_parser|code parser]]: extract strings to be translated from source code and generate .po file
* [[Internationalization/translation function|translation function]]: map hard-coded strings to translated strings ''(Martino)''
** find a solution with templates
** globality: how to implement, the object should be shared between all modules (see [[Internationalization/class_structure|class structure]])
** find out how to use plurals (see [[Internationalization/plural_forms|plural forms]])
* collaborate with the [[Vision2_and_Unicode]] team
* Test unicode support in Vision2. [See test application in the [https://eiffelsoftware.origo.ethz.ch/svn/es/branches/soft-arch/Src/library/i18n/example/ SVN repository]]
* compile a [[Internationalization/features list|list of basic features]] to provide (e.g. date/time format, system locale) ''[deferred]''

= Possible future developments =

* collaborate with the [[ESWizard]] team. Create wizards for programms with translation facilities.

=Relevant Links=
[http://www.debian.org/doc/manuals/intro-i18n/ Introduction to i18n]

==What other people have done==
[http://doc.trolltech.com/4.1/i18n.html internationalisation with QT]

[http://oss.erdfunkstelle.de/kde-i18n/tiki-index.php?page=miniHowtoGui howto for internationalisation of KDE programs ]

[http://l10n.kde.org/docs/translation-howto/ another KDE howto (doesn't Gnome do any internationalisation?)]

=Team=

Everyone interested in this project is welcome to [http://origo.ethz.ch/cgi-bin/mailman/listinfo/es-i18n join our mailinglist] es-i18n@origo.ethz.ch

* Project Leader: [[User:Carlo|Carlo Vanini]]
* [[User:leo| Leo Fellmann]]
* [[User:kiwi| Ivano Somaini]]
* [[User:murbi|Andreas Murbach]]
* [[User:etienner|Etienne Reichenbach]]
* [[User:hong |Hong Zhang]]
* [[User:cconti | Christian Conti]]
* [[User:Trosim |Martino Trosi]]
* [[User:Schoelle| Bernd Schoeller]]

Internationalization/code parser

2006-06-03T06:14:02Z

Etienner:

[[Category:Internationalization]]

page moved to [[Internationalization/mo parser|mo parser]] because content didn't match title. --[[User:Carlo|Carlo]] 18:23, 7 May 2006 (CEST)

Internationalization/file format

2006-06-03T06:13:44Z

Etienner:

[[Category:Internationalization]]

==Summary==

Here [[Internationalization|we]] evaluate various file formats used for the translation of programs. For the moment we are considering:

* XML
* po
* [http://transolution.python-hosting.com xliff] (good description in this homepage)
* create an own format

==PO Files==

===Format of PO files===

A PO file has an entry for each string that has to be translated. There are two kind of them, a "normal" one and one that involves plural forms.

====Normal entry====

Here is the general structure of a "normal" entry:

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string
msgstr translated-string

Where the ''translator-comments'' are created and maintained exclusively by the translator, this comments have some white space immediately following the #. The other comments are created by the program that created the PO file.
After the special comment "#," there can be some ''flags'', as ''fuzzy'' shows that the msgstr string might not be a correct translation, i.e. the translator is not sure of his work.
The 'untranslated-string' is the untranslated string as it appears in the original program source. The ''translated-string'' is (as the name suggests) the translated string, if there is no translation it is an empty string.

====Plural form entry====

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

===Supported character encodings===

character encodings that can be used are limited to those supported by both GNU libc and GNU libiconv. These are: ASCII, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-13, ISO-8859-15, KOI8-R, KOI8-U, CP850, CP866, CP874, CP932, CP949, CP950, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, GB2312, EUC-JP, EUC-KR, EUC-TW, BIG5, BIG5-HKSCS, GBK, GB18030, SHIFT_JIS, JOHAB, TIS-620, VISCII, UTF-8.

I think they are a lot...

===Po Editors===

* poEdit
* KBabel
* Gtranslator
* LocFactoryEditor (XLIFF and PO editor for Mac OSX)

===Positive aspects===

* Powerful plural handling
* Format created for translation purpose
* Easy for humans to read
* Used by gettext, kbabel, rosetta and many other programs
* Support and elaboration tools for almost all plattforms

===Negative aspects===

* We have to write a PO parser

==XML==

===Format of XML===

XML is used for example by Trolltech for their .ts files. Here an example of .ts file, generated my lupdate (a tool made by trolltech that extracts translatable text from the C++ source code of the Qt application, see [[Internationalization/tool evaluation|here]] for further information):

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation type="unfinished"></translation>
</message>
</context>
</TS>

And after the translation (for example with Qt Linguist) it would look like this:

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation>i20e=Internazionalizzazione</translation>
</message>
</context>
</TS>

===Positive aspects===

* full support for unicode character encodings
* There is already a parser in the EiffelBase
* In trolltech's opinion it's a human readable text

===Negative aspects===

* Not everybody knows it
* Microsoft seeks XML-related patents that could restrict the use of XML (there should be a "Very negative aspect" section)
:Thank you for that link, but the Microsoft Patents apply only on the Office dialect of XML, which it uses to save its own documents... they DON'T apply to the XML standard and general use (as we would create our format). [[User:Trosim|Trosim]]
* In my opinion it's not a human readable text (Fortunately not all human beings are Computer scientists)

==New Format==

===Format of our Format===

* It doesn't exist yet, so we don't know how it looks like.
* We could give our own extension to the file format for example .et (eiffel translation) or .babe (babylon eiffel) or .eint (eiffel i18n) ... (''huge'' advantage)

===Positive aspects===

* Free to do what we want
* We don't have to care about licenses
* Possibility to make it the best human readable format
* So that we can say that we invented a new file format
* Better integrated and consistent with eiffel syntax

===Negative aspects===

* A new format? Why should we be different?
* Do more work as needed (there are already good formats)
* Long time until it becomes famous

==Conclusions==

Our decision is to use the po file format.

==References==

* [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html Gettext manual (for PO files)]
* [http://librarian.launchpad.net/2395419/it.po Example of a PO file (from rosetta)]
* [http://news.com.com/2100-1013_3-5146581.html Microsoft and XML]
* [http://en.wikipedia.org/wiki/XML#Quick_syntax_tour XML syntax description on Wikipedia]
* [http://doc.trolltech.com/4.1/i18n.html Trolltech i18n]
* [http://translate.sourceforge.net/wiki/ open source i18n and l10n project]

Internationalization/mo parser

2006-06-03T06:13:23Z

Etienner:

[[Category:Internationalization]]

== Summary ==
That's what this part of the project should achieve:
* reading and parsing of MO files containing the strings and their translations
* organize the object collection in an incremental way: don't load the whole file if it's not needed
* give a simple interface to the localization class, so that the strings can be printed out without too much efforts

==What is a mo file==

MO stands for '''M'''achine '''O'''bject, as the name suggests, they are meant to be read by programs, and are binary in nature. The GNU program that creates them out of a PO file, is [http://www.scit.wlv.ac.uk/cgi-bin/mansec?1+msgfmt ''msgfmt''].

== Reading and parsing ==

=== Parser structure ===

I'll propose the class structure of the parser, later.

=== MO file structure ===

As reported from the [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html#SEC136 gettext manual].

byte
+------------------------------------------+
0 | magic number = 0x950412de |
| |
4 | file format revision = 0 |
| |
8 | number of strings | == N
| |
12 | offset of table with original strings | == O
| |
16 | offset of table with translation strings | == T
| |
20 | size of hashing table | == S
| |
24 | offset of hashing table | == H
| |
. .
. (possibly more entries later) .
. .
| |
O | length & offset 0th string ----------------.
O + 8 | length & offset 1st string ------------------.
... ... | |
O + ((N-1)*8) | length & offset (N-1)th string | | |
| | | |
T | length & offset 0th translation ---------------.
T + 8 | length & offset 1st translation -----------------.
... ... | | | |
T + ((N-1)*8) | length & offset (N-1)th translation | | | | |
| | | | | |
H | start hash table | | | | |
... ... | | | |
H + S * 4 | end hash table | | | | |
| | | | | |
| NUL terminated 0th string <----------------' | | |
| | | | |
| NUL terminated 1st string <------------------' | |
| | | |
... ... | |
| | | |
| NUL terminated 0th translation <---------------' |
| | |
| NUL terminated 1st translation <-----------------'
| |
... ...
| |
+------------------------------------------+
====Magic number====

The magic number signals GNU MO files, it is stored in the byte order of the generating machine, so the magic number really is two numbers: 0x950412de and 0xde120495

====Tables====

In the tables, each string descriptor uses two 32 bits integers, the first for the string length, and the second for the offset of the string in the MO file, counting in bytes from the start of the file. The first table contains descriptors for the original strings, and is sorted in lexicographical order. The second table contains descriptors for the translated strings, and is parallel to the first table: to find the corresponding translation one has to access the array slot in the second array with the same index.

====Hash table====

The hash table is not contained in al Mo files, in this cases the size S is zero. It is sometimes better to not include the hash table because a precomputed hashing table takes disk space, and does not win that much speed. The hash table contains indices to the sorted array of strings in the MO file.

==Hash Function==

Here the description of the hash function used by gettext. I've taken it from it's source code [ftp://mirrors.kernel.org/gnu/gettext/gettext-0.14.5.tar.gz gettext-0.14.5].

Each string has an associate hashing value V computed by a fixed function (see below). To locate the string, open addressing with double hashing is used. The first index will be V % S, where S is the size of the hashing table. If no entry is found, iterating with a second, independent hashing function takes place. This second value will be:
1 + V % (S - 2).
The approximate number of probes will be

:for unsuccessful search: (1 - N / S) ^ -1
:for successful search: - (N / S) ^ -1 * ln (1 - N / S)
::where N is the number of keys.

If we now choose S to be the next prime bigger than 4 / 3 * N,
we get the values
:::4 and 1.85 resp.
Because unsuccessful searches are unlikely this is a good value.
Formulas: [Knuth, The Art of Computer Programming, Volume 3, Sorting and Searching, 1973, Addison Wesley]

The hash function used to convert stings to an integer, is the so called ''hashpjw'' function by P.J. Weinberger [see Aho/Sethi/Ullman, COMPILERS: Principles, Techniques and Tools, 1986, 1987 Bell Telephone Laboratories, Inc.]

The C code of the function comes from the source code of gettext (./gettext-0.14.5/gettext-runtime/intl/hash-string.h):

<code>[c,N]
#define HASHWORDBITS 32
hash_string (const char *str_param) {
unsigned long int hval, g;
const char *str = str_param;

/* Compute the hash value for the given string. */
hval = 0
while (*str != '\0'
{
hval <<= 4;
hval += (unsigned char) *str++;
g = hval & ((unsigned long int) 0xf << (HASHWORDBITS - 4));
if (g != 0
{
hval ^= g >> (HASHWORDBITS - 8);
hval ^= g;
}
}
return hval;
}
</code>

Here is the translated function in Eiffel:
<code>[eiffel,N]
hash_string (a_string: STRING): INTEGER is
-- Compute the hash value for the given string
require
valid_string: a_string /= Void
local
position: INTEGER
l_result, g: NATURAL_32
do
from
position := 1
invariant
position >= 1
position <= a_string.count + 1
variant
a_string.count + 1 - position
until
position > a_string.count
loop
l_result := l_result |<< 4
l_result := l_result + a_string.code(position)
g := l_result & ({NATURAL_32} 0xf |<< 28)
if g /= 0 then
l_result := l_result.bit_xor(g |>> 24)
l_result := l_result.bit_xor(g)
end
position := position + 1
end
Result := l_result.as_integer_32
end
</code>

Internationalization/plural forms

2006-06-03T06:11:46Z

Etienner:

[[Category:Internationalization]]

The information about the plural form selection has to be stored in the header entry of the PO file (the one with the empty msgid string). The plural form information looks like this:
::Plural-Forms: nplurals=2; plural=n == 1 ? 0 : 1;

The ''nplurals'' value must be a decimal number which specifies how many different plural forms exist for this language. The string following plural is an expression which is using the C language syntax. Exceptions are that no negative numbers are allowed, numbers must be decimal, and the only variable allowed is ''n''. This expression will be evaluated whenever one of the [[internationalization/translation_function|translation function]] for a "plural form" is called. The numeric value passed to these functions is then substituted for all uses of the variable n in the expression. The resulting value then must be greater or equal to zero and smaller than the value given as the value of ''nplurals''.

==Rules==
The (at this point) known rules are:
===Only one form===

Some languages only require one single form. There is no distinction between the singular and plural form. An appropriate header entry would look like this:

:Plural-Forms: nplurals=1; plural=0;

:Languages with this property include:
:*Finno-Ugric family
:*Hungarian
:*Asian family
:*Japanese, Korean
:*Turkic/Altaic family
:*Turkish

===Two forms, singular used for one only===

This is the form used in most existing programs since it is what English is using. A header entry would look like this:

:Plural-Forms: nplurals=2; plural=n != 1;
:::(Note: this uses the feature of C expressions that boolean expressions have to value zero or one.)

:Languages with this property include:
:*Germanic family
:*Danish, Dutch, English, German, Norwegian, Swedish
:*Finno-Ugric family
:*Estonian, Finnish
:*Latin/Greek family
:*Greek
:*Semitic family
:*Hebrew
:*Romanic family
:*Italian, Portuguese, Spanish
:*Artificial
:*Esperanto

===Two forms, singular used for zero and one===

Exceptional case in the language family. The header entry would be:

:Plural-Forms: nplurals=2; plural=n>1;

:Languages with this property include:
:*Romanic family
:*French, Brazilian Portuguese

===Three forms, special case for zero===

The header entry would be:

:Plural-Forms: nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n != 0 ? 1 : 2;

:Languages with this property include:
:*Baltic family
:*Latvian

===Three forms, special cases for one and two===
The header entry would be:

:Plural-Forms: nplurals=3; plural=n==1 ? 0 : n==2 ? 1 : 2;

:Languages with this property include:
:*Celtic
:*Gaeilge (Irish)

===Three forms, special case for numbers ending in 1[2-9]===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n%10==1 && n%100!=11 ? 0 : \
::::::::n%10>=2 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Baltic family
:*Lithuanian

===Three forms, special cases for numbers ending in 1 and 2, 3, 4, except those ending in 1[1-4]===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n%10==1 && n%100!=11 ? 0 : \
::::n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Slavic family
:*Croatian, Czech, Russian, Slovak, Ukrainian

===Three forms, special case for one and some numbers ending in 2, 3, or 4===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n==1 ? 0 : \
::::n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Slavic family
:*Polish

===Four forms, special case for one and all numbers ending in 02, 03, or 04===

The header entry would look like this:

:Plural-Forms: nplurals=4; \
::plural=n%100==1 ? 0 : n%100==2 ? 1 : n%100==3 || n%100==4 ? 2 : 3;

:Languages with this property include:
:*Slavic family
:*Slovenian

Internationalization/tool evaluation

2006-06-03T06:11:24Z

Etienner:

[[Category:Internationalization]]

In this section it will be discussed which tool should be taken for the i18n work.

= What should the tools be able to do =
* Search the source code for strings and other things that needs to be changed
* Create a file containing these results
* Enable the change of language and format of Eiffel

= Tasks =

The tools we should analyse have several tasks:
* find the strings into the source code and extract them
* find "information" about specific formats (time, measure units,...) in the code
* give the possibility to translate in a friendly environment
* replace the string with a function(string) construct

Here you'll find the advantages and drawbacks of every tool we evaluate.

== Tools to evaluate ==
Here is the list of existing tools we are now evaluating:

=== GNU gettext ===
This set of tool is funded by the GNU community and is meant to help internationalize the C-Programs (and also C++, ObjectiveC, Pascal, PO, Python, Lisp, EmacsLisp, librep, Java, awk, YCP, Tcl, RST, Glade). As we are dealing with Eiffel this can be a bit harder to implement, but some tools can still be used.

Advantages
* has tools to translate
* has tools to merge new strings into the actual catalog
* independent from the platform
Drawbacks
* uses only PO files (we should write a parser)
* has no runtime library for Eiffel

=== poEdit ===
Yet another PO catalog editor.

Advantages
* has tools to translate
* independent from the platform
Drawbacks
* uses only PO files (we should write a parser)
* has no runtime library for Eiffel

[[http://www.poedit.org/ Visit the website for more information.]]

=== KBabel ===
This is the tool used by the KDE community to translate the interface, it works with gettext and PO files.
From their [http://kbabel.kde.org/ website]:
"KBabel is a set of tools for editing and managing gettext PO files. Main part is a powerful and comfortable PO file editor which features full navigation capabilities, full editing functionality, possibility to search for translations in different dictionaries, spell and syntax checking, showing diffs and many more. Also included is a "Catalog Manager", a file manager view which helps keeping an overview of PO files. Last but not least it includes a standalone dictionary application as an additional possibility to access KBabel's powerful dictionaries. KBabel will help you to translate fast and also keep consistent translations."

Features of the translation tool:
* Support for GNU gettext PO files (including plural forms) and Qt Linguist catalogs
* Experimental support for XLIFF 1.0 format
* Support for multiple translation projects
* Capability to open multiple files (or multiple views of the same file)
* Full editing functionality, accessible through the customizable graphic user interface as well as through user definable keyboard shortcuts
* Powerful spell checking functionality
* Capability to show diffs to older messages
* Full navigation capabilities (such as go to next fuzzy or untranslated string)
* Capability to save and read files in unicode encoding (utf-8)
* Unlimited undo capability
* Syntax highlighting
* Automatic file header updates
* Automatic change of "fuzzy" status of translated messages
* Support for easy insertion of tags and URLs
* Validation and highlighting of tags and XML entities
* A plugin framework for dictionaries, such as po compendium files, for consistency checks or translation suggestions
* A "rough translation" function to initialize untranslated messages with suggestion from a dictionary
* Automatic syntax check with msgfmt when saving and if an error occured easy navigation to messages, which contain errors
* Full Drag & Drop - support
* Configurable fonts for message editor
* Various methods to "see" whitespaces at end of lines
* Various methods to check consistency of translated messages, like comparing printf and Qt arguments in msgid and msgstr
* Quick overview over context in the po file
* Showing source code by references in message comments
* Sending the file using email
* Bookmarks for messages in a file
* Character selection tool integration
* A plugin framework for validation tools for consistency checks

Features of the Catalog Manager:
* File manager view for KDE's l10n module (or similarly structured) directories, which shows the present status of all PO files: if they are in need of a revision or not, how many fuzzies and untranslated strings are included etc. This view is always automatically updated and reflects all changes done to the files, including changes by programs other than KBabel.
* Integrated basic CVS and SVN support
* Various file open mechanisms for editing in KBabel: use Drag & Drop, double click, keyboard or context menu
* "Mark files" function (e.g. to identify POs that are in the responsibility of other translators)
* Powerful navigation using PO file statistics
* Automatic comparisons and statistics of POT and PO files for a quick overview which and how many files are translated (or not) and which files may be obsolete
* Syntax check (msgfmt --statistics) for existing files to control if the translated files will compile and, accordingly, work when distributed
* Free configurable commands, that can be executed from the Catalog Manager's context menu.
* Search/Replace functions in multiple files at once.
* Spellchecking of multiple files at once.
* Sending number of files using email.
* Doing "rough translation" for multiple files at once.

Notes
* under GNU GPL
* part of the kdesdk module
* it also has KBabelDict, plugin-based dictionaries (web-based but can be downloaded as a plain txt file), needs some Perl modules.
* runs only on linux

Info from the official website of KBabel
[[http://l10n.kde.org/ click here for more]]

On their website infos about Kartouche, a web-based php tool to help translation, is also available.

From the infos available about this tool it looks like it's ideal to reach our goal, it has all we need and more.

Advantages:
* Nice and easy to use interface
* Helps A LOT the translator
* Tool supported by the KBabel community
* GNU GPL
* Supports unicode
* Set of dictionaries

Drawbacks:
* Runs on Linux systems only
* May need some work to adapt it for Eiffel

=== KTranslator ===
This tool helps the translation of single words, independently from the used application.

Advantages:
* Application independant, used while running the program

Drawbacks:
* Uses popups to display the translation of the word
* It works exclusively with dictionaries
* It isn't really a tool that can do big work of translation

[http://ktranslator.sourceforge.net/index.html More]

=== Other tools to watch ===
* xgettext
* poxml (Definition: tools for using PO-files to translate DocBook XML files This is a collection of tools that facilitate translating DocBook XML files using gettext message files (PO-files).)
* [http://po4a.alioth.debian.org/ po4a] (The po4a (po for anything) project goal is to ease translations (and more interestingly, the maintenance of translations) using gettext tools on areas where they were not expected like documentation.)
* xml2po
* [http://nlso-objects.sourceforge.net/ NLSO] (for Windows)
* Kartouche

=== Conclusions ===

== Writing our own tools ==
That could be a good idea to write our own tools to do these tasks, in particular for a distributed translation (everybody can help on internet).
With this I mean a little website that allows us to do the actual translation directly via the web, and it should automatically create the new version of the language file to be downloaded from all the users.

Another thing that this tool can do for us is scanning the entire source code looking for new translations to be done, and propose them to the users... can be interesting because we than only have to click on a link and all the work (not the translations) is done for us.

Advantages
* independent from the architecture
* independent from the language file format
Drawbacks
* can be a lot of work

= Problems, Ideas, Further Development =
some problems/ideas that may go with the other parts of this project as well:
* The i18n can be put online as a database that everyone can update (thus helping translation in many more languages)
After the user has chosen a language the entries still untranslated will be displayed so that the user can insert his own version of the translated message
There may be the problem of some users who may have fun writing bad, strange, crazy, nonsense sentences, thus a solution is needed since it clearly can't be checked by the project team
* The main database can be updated as soon as something new for eiffel comes out (by updated here is meant simply adding the new messages, not their translations)
* A 2nd database can be placed on the user's pc and autoupdate Eiffel and the additional content one may have when needed (when a new version of Eiffel or a new tool is available for example), with this system one doesn't need to download every time a new file containing some information already downloaded before
* Unicode problem (maybe a collaboration with the Unicode group can help)

Internationalization/translation function

2006-06-03T06:11:00Z

Etienner:

[[Category:Internationalization]]

==Introduction==

The problem for the translation is that we need a way to translate three different types of string:

* normal strings
<code>[eiffel,N] "A normal string"</code>
* composed string
<code>[eiffel,N] "A string composed by "+count.out+" strings, like this or "+a_string+"%N"</code>
* strings with plural form
<code>[eiffel,N]
if n=1 then
Result := "a string"
else
Result := "strings"
end
</code>

Another problem is that not all languages have the same rules for plural forms, see below and [[internationalization/plural_forms|here]] for more details.

==Possible solution==

I [[Talk:Internationalization/translation_function|'''propose''']] a solution, with four functions (I've got little fantasy for the function names):

<code>[eiffel,N]
i18n (a_string : STRING) : STRING
--Function for the translation of normal strings

i18n_pl (strings : TUPLE[STRING]; form : INTEGER) : STRING
--Function for the translation of normal strings
-- with plural form

i18n_comp (a_string : STRING; args : TUPLE) : STRING
--Function for the translation of composit strings

i18n_comp_pl (strings : TUPLE[STRING]; args : TUPLE; form : INTEGER) : STRING
--Function for the translation of composit
-- strings with plural forms
</code>

With this solution, the programmer should write composit strings as (or in a similar way) he would do it for the C ''printf'' function. The composit string above would be somthing like this:

<code>[eiffel,N] i18n_comp("A string composed by %a1 strings, like this or %a2%N", [count, a_string])
</code>

where ''%a1'' is the place where the first string in the argument tuple (here ''count'') has to go. I've [[Talk:Internationalization/translation_function|'''choosen''']] that the items in the argument tuple are of type ANY, to insert them in the string we have to apply the functon ''out'' to all items.

===An example===

A complete example of how it could work.

----
Piece of code without internationalization:

<code>[eiffel,N]

[...]
if n = 1
io.put_string("Remove file "+file_name.out+"?%N")
else
io.put_string ("Delete following files?"+list.out+" there are "
+list.count.out+"files%N")
end
[...]

</code>

----

Piece of code with internationalization:

<code>[eiffel,N]

[...]
io.put_string(i18n_comp_pl(["Remove file %a1?%N",
"Delete following files? %a2 there are %a3 files%N"],
[file_name, list, list_count], n))
[...]
</code>

the content of the PO file would be:

#: A comment
msgid "Remove file %a1?%N"
msgid_plural "Delete following files? %a2 there are %a3 files%N"
msgstr[0] "Rimuovere il file %a1?"
msgstr[1] "Cancellare i seguenti file? %a2 ce ne sono %a3%N"

You might ask yourself, ''why a function for all the pluralforms''?

The number of plural forms differ between languages. This is somewhat surprising for those who only have experiences with Romanic and Germanic languages since here the number is the same (there are two).
But other language families have only one form or many forms. For example,

In Polish the translation of file is plik, and the plural forms are:

:2,3,4 pliki
:5-21 pliko'w
:22-24 pliki
:25-31 pliko'w
:and so on...

for more information about this topic go [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html#SEC150 here]

==Additional things==

A nice thing whe could do, when parsing the source code of a program to internationalize, is to recognize comments about the string.
For example:

<code>[eiffel,N]

[...]
io.put_string(i18n_comp_pl(["Remove file %a1?%N",
"Delete following files? %a2 there are %a3 files%N"],
[file_name, list, list_count], list_count))
--A comment of the author
[...]
</code>

woud then look like this in the PO file:

#. A comment of the author
#. Function name, class name, file path
#. %a1 = file_name
#. %a2 = list
#. %a3 = list_count
msgid "Remove file %a1?%N"
msgid_plural "Delete following files? %a2 there are %a3 files%N"
msgstr[0] "Rimuovere il file %a1?"
msgstr[1] "Cancellare i seguenti file? %a2 ce ne sono %a3%N"

Internationalization/plural forms

2006-05-31T16:51:30Z

Etienner: added page to our category

[[Category:Projects]]
[[Category:Internationalization]]

The information about the plural form selection has to be stored in the header entry of the PO file (the one with the empty msgid string). The plural form information looks like this:
::Plural-Forms: nplurals=2; plural=n == 1 ? 0 : 1;

The ''nplurals'' value must be a decimal number which specifies how many different plural forms exist for this language. The string following plural is an expression which is using the C language syntax. Exceptions are that no negative numbers are allowed, numbers must be decimal, and the only variable allowed is ''n''. This expression will be evaluated whenever one of the [[internationalization/translation_function|translation function]] for a "plural form" is called. The numeric value passed to these functions is then substituted for all uses of the variable n in the expression. The resulting value then must be greater or equal to zero and smaller than the value given as the value of ''nplurals''.

==Rules==
The (at this point) known rules are:
===Only one form===

Some languages only require one single form. There is no distinction between the singular and plural form. An appropriate header entry would look like this:

:Plural-Forms: nplurals=1; plural=0;

:Languages with this property include:
:*Finno-Ugric family
:*Hungarian
:*Asian family
:*Japanese, Korean
:*Turkic/Altaic family
:*Turkish

===Two forms, singular used for one only===

This is the form used in most existing programs since it is what English is using. A header entry would look like this:

:Plural-Forms: nplurals=2; plural=n != 1;
:::(Note: this uses the feature of C expressions that boolean expressions have to value zero or one.)

:Languages with this property include:
:*Germanic family
:*Danish, Dutch, English, German, Norwegian, Swedish
:*Finno-Ugric family
:*Estonian, Finnish
:*Latin/Greek family
:*Greek
:*Semitic family
:*Hebrew
:*Romanic family
:*Italian, Portuguese, Spanish
:*Artificial
:*Esperanto

===Two forms, singular used for zero and one===

Exceptional case in the language family. The header entry would be:

:Plural-Forms: nplurals=2; plural=n>1;

:Languages with this property include:
:*Romanic family
:*French, Brazilian Portuguese

===Three forms, special case for zero===

The header entry would be:

:Plural-Forms: nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n != 0 ? 1 : 2;

:Languages with this property include:
:*Baltic family
:*Latvian

===Three forms, special cases for one and two===
The header entry would be:

:Plural-Forms: nplurals=3; plural=n==1 ? 0 : n==2 ? 1 : 2;

:Languages with this property include:
:*Celtic
:*Gaeilge (Irish)

===Three forms, special case for numbers ending in 1[2-9]===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n%10==1 && n%100!=11 ? 0 : \
::::::::n%10>=2 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Baltic family
:*Lithuanian

===Three forms, special cases for numbers ending in 1 and 2, 3, 4, except those ending in 1[1-4]===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n%10==1 && n%100!=11 ? 0 : \
::::n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Slavic family
:*Croatian, Czech, Russian, Slovak, Ukrainian

===Three forms, special case for one and some numbers ending in 2, 3, or 4===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n==1 ? 0 : \
::::n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Slavic family
:*Polish

===Four forms, special case for one and all numbers ending in 02, 03, or 04===

The header entry would look like this:

:Plural-Forms: nplurals=4; \
::plural=n%100==1 ? 0 : n%100==2 ? 1 : n%100==3 || n%100==4 ? 2 : 3;

:Languages with this property include:
:*Slavic family
:*Slovenian

Internationalization/translation function

2006-05-31T16:51:12Z

Etienner: added page to our category

[[Category:Projects]]
[[Category:Internationalization]]

==Introduction==

The problem for the translation is that we need a way to translate tree different types of string:

* normal strings
<code>[eiffel,N] "A normal string"</code>
* composed string
<code>[eiffel,N] "A string composed by "+count.out+" strings, like this or "+a_string+"%N"</code>
* strings with plural form
<code>[eiffel,N]
if n=1 then
Result := "a string"
else
Result := "strings"
end
</code>

An other problem is that not all languages have the same rule for plural forms, see below and [[internationalization/plural_forms|here]] for more details.

==Possible solution==

I [[Talk:Internationalization/translation_function|'''propose''']] a solution, with four functions (I've got little fantasy for the function names):

<code>[eiffel,N]
i18n (a_string : STRING) : STRING
--Function for the translation of normal strings

i18n_pl (strings : TUPLE[STRING]; form : INTEGER) : STRING
--Function for the translation of normal strings
-- with plural form

i18n_comp (a_string : STRING; args : TUPLE) : STRING
--Function for the translation of composit strings

i18n_comp_pl (strings : TUPLE[STRING]; args : TUPLE; form : INTEGER) : STRING
--Function for the translation of composit
-- strings with plural forms
</code>

With this solution, the programmer should write composit strings as (or in a similar way) he would do it for the C ''printf'' function. The composit string above would be somthing like this:

<code>[eiffel,N] i18n_comp("A string composed by %a1 strings, like this or %a2%N", [count, a_string])
</code>

where ''%a1'' is the place where the first string in the argument tuple (here ''count'') has to go. I've [[Talk:Internationalization/translation_function|'''choosen''']] that the items in the argument tuple are of type ANY, to insert them in the string we have to apply the functon ''out'' to all items.

===An example===

A complete example of how it could work.

----
Piece of code without internationalization:

<code>[eiffel,N]

[...]
if n = 1
io.put_string("Remove file "+file_name.out+"?%N")
else
io.put_string ("Delete following files?"+list.out+" there are "
+list.count.out+"files%N")
end
[...]

</code>

----

Piece of code with internationalization:

<code>[eiffel,N]

[...]
io.put_string(["Remove file %a1?%N",
"Delete following files? %a2 there are %a3 files%N"],
[file_name, list, list_count], n)
[...]
</code>

the content of the PO file would be:

#: A comment
msgid "Remove file %a1?%N"
msgid_plural "Delete following files? %a2 there are %a3 files%N"
msgstr[0] "Rimuovere il file %a1?"
msgstr[1] "Cancellare i seguenti file? %a2 ce ne sono %a3%N"

You might ask yourself, ''why a function for all the pluralforms''?

The number of plural forms differ between languages. This is somewhat surprising for those who only have experiences with Romanic and Germanic languages since here the number is the same (there are two).
But other language families have only one form or many forms. For example,

In Polish the translation of file is plik, and the plural forms are:

:2,3,4 pliki
:5-21 pliko'w
:22-24 pliki
:25-31 pliko'w
:and so on...

for more information about this topic go [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html#SEC150 here]

==Additional things==

A nice thing whe could do, when parsing the source code of a program to internationalize, is to recognize comments about the string.
For example:

<code>[eiffel,N]

[...]
io.put_string(["Remove file %a1?%N",
"Delete following files? %a2 there are %a3 files%N"],
[file_name, list, list_count], n)
--A comment of the author
[...]
</code>

woud then look like this in the PO file:

#. A comment of the author
#. Function name, class name, file path
#. %a1 = file_name
#. %a2 = list
#. %a3 = list_count
msgid "Remove file %a1?%N"
msgid_plural "Delete following files? %a2 there are %a3 files%N"
msgstr[0] "Rimuovere il file %a1?"
msgstr[1] "Cancellare i seguenti file? %a2 ce ne sono %a3%N"

Internationalization/code parser

2006-05-31T16:50:39Z

Etienner: added page to our category

[[Category:Projects]]
[[Category:Internationalization]]

page moved to [[Internationalization/mo parser|mo parser]] because content didn't match title. --[[User:Carlo|Carlo]] 18:23, 7 May 2006 (CEST)

Internationalization/mo parser

2006-05-31T16:50:27Z

Etienner: added page to our category

[[Category:Projects]]
[[Category:Internationalization]]

== Summary ==
That's what this part of the project should achieve:
* reading and parsing of MO files containing the strings and their translations
* organize the object collection in an incremental way: don't load the whole file if it's not needed
* give a simple interface to the localization class, so that the strings can be printed out without too much efforts

==What is a mo file==

MO stands for '''M'''achine '''O'''bject, as the name suggests, they are meant to be read by programs, and are binary in nature. The GNU program that creates them out of a PO file, is [http://www.scit.wlv.ac.uk/cgi-bin/mansec?1+msgfmt ''msgfmt''].

== Reading and parsing ==

=== Parser structure ===

I'll propose the class structure of the parser, later.

=== MO file structure ===

As reported from the [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html#SEC136 gettext manual].

byte
+------------------------------------------+
0 | magic number = 0x950412de |
| |
4 | file format revision = 0 |
| |
8 | number of strings | == N
| |
12 | offset of table with original strings | == O
| |
16 | offset of table with translation strings | == T
| |
20 | size of hashing table | == S
| |
24 | offset of hashing table | == H
| |
. .
. (possibly more entries later) .
. .
| |
O | length & offset 0th string ----------------.
O + 8 | length & offset 1st string ------------------.
... ... | |
O + ((N-1)*8) | length & offset (N-1)th string | | |
| | | |
T | length & offset 0th translation ---------------.
T + 8 | length & offset 1st translation -----------------.
... ... | | | |
T + ((N-1)*8) | length & offset (N-1)th translation | | | | |
| | | | | |
H | start hash table | | | | |
... ... | | | |
H + S * 4 | end hash table | | | | |
| | | | | |
| NUL terminated 0th string <----------------' | | |
| | | | |
| NUL terminated 1st string <------------------' | |
| | | |
... ... | |
| | | |
| NUL terminated 0th translation <---------------' |
| | |
| NUL terminated 1st translation <-----------------'
| |
... ...
| |
+------------------------------------------+
====Magic number====

The magic number signals GNU MO files, it is stored in the byte order of the generating machine, so the magic number really is two numbers: 0x950412de and 0xde120495

====Tables====

In the tables, each string descriptor uses two 32 bits integers, the first for the string length, and the second for the offset of the string in the MO file, counting in bytes from the start of the file. The first table contains descriptors for the original strings, and is sorted in lexicographical order. The second table contains descriptors for the translated strings, and is parallel to the first table: to find the corresponding translation one has to access the array slot in the second array with the same index.

====Hash table====

The hash table is not contained in al Mo files, in this cases the size S is zero. It is sometimes better to not include the hash table because a precomputed hashing table takes disk space, and does not win that much speed. The hash table contains indices to the sorted array of strings in the MO file.

==Hash Function==

Here the description of the hash function used by gettext. I've taken it from it's source code [ftp://mirrors.kernel.org/gnu/gettext/gettext-0.14.5.tar.gz gettext-0.14.5].

Each string has an associate hashing value V computed by a fixed function (see below). To locate the string, open addressing with double hashing is used. The first index will be V % S, where S is the size of the hashing table. If no entry is found, iterating with a second, independent hashing function takes place. This second value will be:
1 + V % (S - 2).
The approximate number of probes will be

:for unsuccessful search: (1 - N / S) ^ -1
:for successful search: - (N / S) ^ -1 * ln (1 - N / S)
::where N is the number of keys.

If we now choose S to be the next prime bigger than 4 / 3 * N,
we get the values
:::4 and 1.85 resp.
Because unsuccessful searches are unlikely this is a good value.
Formulas: [Knuth, The Art of Computer Programming, Volume 3, Sorting and Searching, 1973, Addison Wesley]

The hash function used to convert stings to an integer, is the so called ''hashpjw'' function by P.J. Weinberger [see Aho/Sethi/Ullman, COMPILERS: Principles, Techniques and Tools, 1986, 1987 Bell Telephone Laboratories, Inc.]

The C code of the function comes from the source code of gettext (./gettext-0.14.5/gettext-runtime/intl/hash-string.h):

<code>[c,N]
#define HASHWORDBITS 32
hash_string (const char *str_param) {
unsigned long int hval, g;
const char *str = str_param;

/* Compute the hash value for the given string. */
hval = 0
while (*str != '\0'
{
hval <<= 4;
hval += (unsigned char) *str++;
g = hval & ((unsigned long int) 0xf << (HASHWORDBITS - 4));
if (g != 0
{
hval ^= g >> (HASHWORDBITS - 8);
hval ^= g;
}
}
return hval;
}
</code>

Here is the translated function in Eiffel:
<code>[eiffel,N]
hash_string (a_string: STRING): INTEGER is
-- Compute the hash value for the given string
require
valid_string: a_string /= Void
local
position: INTEGER
l_result, g: NATURAL_32
do
from
position := 1
invariant
position >= 1
position <= a_string.count + 1
variant
a_string.count + 1 - position
until
position > a_string.count
loop
l_result := l_result |<< 4
l_result := l_result + a_string.code(position)
g := l_result & ({NATURAL_32} 0xf |<< 28)
if g /= 0 then
l_result := l_result.bit_xor(g |>> 24)
l_result := l_result.bit_xor(g)
end
position := position + 1
end
Result := l_result.as_integer_32
end
</code>

Internationalization/tool evaluation

2006-05-31T16:50:13Z

Etienner: added page to our category

[[Category:Projects]]
[[Category:Internationalization]]

In this section it will be discussed which tool should be taken for the i18n work.

= What should the tools be able to do =
* Search the source code for strings and other things that needs to be changed
* Create a file containing these results
* Enable the change of language and format of Eiffel

= Tasks =

The tools we should analyse have several tasks:
* find the strings into the source code and extract them
* find "information" about specific formats (time, measure units,...) in the code
* give the possibility to translate in a friendly environment
* replace the string with a function(string) construct

Here you'll find the advantages and drawbacks of every tool we evaluate.

== Tools to evaluate ==
Here is the list of existing tools we are now evaluating:

=== GNU gettext ===
This set of tool is funded by the GNU community and is meant to help internationalize the C-Programs (and also C++, ObjectiveC, Pascal, PO, Python, Lisp, EmacsLisp, librep, Java, awk, YCP, Tcl, RST, Glade). As we are dealing with Eiffel this can be a bit harder to implement, but some tools can still be used.

Advantages
* has tools to translate
* has tools to merge new strings into the actual catalog
* independent from the platform
Drawbacks
* uses only PO files (we should write a parser)
* has no runtime library for Eiffel

=== poEdit ===
Yet another PO catalog editor.

Advantages
* has tools to translate
* independent from the platform
Drawbacks
* uses only PO files (we should write a parser)
* has no runtime library for Eiffel

[[http://www.poedit.org/ Visit the website for more information.]]

=== KBabel ===
This is the tool used by the KDE community to translate the interface, it works with gettext and PO files.
From their [http://kbabel.kde.org/ website]:
"KBabel is a set of tools for editing and managing gettext PO files. Main part is a powerful and comfortable PO file editor which features full navigation capabilities, full editing functionality, possibility to search for translations in different dictionaries, spell and syntax checking, showing diffs and many more. Also included is a "Catalog Manager", a file manager view which helps keeping an overview of PO files. Last but not least it includes a standalone dictionary application as an additional possibility to access KBabel's powerful dictionaries. KBabel will help you to translate fast and also keep consistent translations."

Features of the translation tool:
* Support for GNU gettext PO files (including plural forms) and Qt Linguist catalogs
* Experimental support for XLIFF 1.0 format
* Support for multiple translation projects
* Capability to open multiple files (or multiple views of the same file)
* Full editing functionality, accessible through the customizable graphic user interface as well as through user definable keyboard shortcuts
* Powerful spell checking functionality
* Capability to show diffs to older messages
* Full navigation capabilities (such as go to next fuzzy or untranslated string)
* Capability to save and read files in unicode encoding (utf-8)
* Unlimited undo capability
* Syntax highlighting
* Automatic file header updates
* Automatic change of "fuzzy" status of translated messages
* Support for easy insertion of tags and URLs
* Validation and highlighting of tags and XML entities
* A plugin framework for dictionaries, such as po compendium files, for consistency checks or translation suggestions
* A "rough translation" function to initialize untranslated messages with suggestion from a dictionary
* Automatic syntax check with msgfmt when saving and if an error occured easy navigation to messages, which contain errors
* Full Drag & Drop - support
* Configurable fonts for message editor
* Various methods to "see" whitespaces at end of lines
* Various methods to check consistency of translated messages, like comparing printf and Qt arguments in msgid and msgstr
* Quick overview over context in the po file
* Showing source code by references in message comments
* Sending the file using email
* Bookmarks for messages in a file
* Character selection tool integration
* A plugin framework for validation tools for consistency checks

Features of the Catalog Manager:
* File manager view for KDE's l10n module (or similarly structured) directories, which shows the present status of all PO files: if they are in need of a revision or not, how many fuzzies and untranslated strings are included etc. This view is always automatically updated and reflects all changes done to the files, including changes by programs other than KBabel.
* Integrated basic CVS and SVN support
* Various file open mechanisms for editing in KBabel: use Drag & Drop, double click, keyboard or context menu
* "Mark files" function (e.g. to identify POs that are in the responsibility of other translators)
* Powerful navigation using PO file statistics
* Automatic comparisons and statistics of POT and PO files for a quick overview which and how many files are translated (or not) and which files may be obsolete
* Syntax check (msgfmt --statistics) for existing files to control if the translated files will compile and, accordingly, work when distributed
* Free configurable commands, that can be executed from the Catalog Manager's context menu.
* Search/Replace functions in multiple files at once.
* Spellchecking of multiple files at once.
* Sending number of files using email.
* Doing "rough translation" for multiple files at once.

Notes
* under GNU GPL
* part of the kdesdk module
* it also has KBabelDict, plugin-based dictionaries (web-based but can be downloaded as a plain txt file), needs some Perl modules.
* runs only on linux

Info from the official website of KBabel
[[http://l10n.kde.org/ click here for more]]

On their website infos about Kartouche, a web-based php tool to help translation, is also available.

From the infos available about this tool it looks like it's ideal to reach our goal, it has all we need and more.

Advantages:
* Nice and easy to use interface
* Helps A LOT the translator
* Tool supported by the KBabel community
* GNU GPL
* Supports unicode
* Set of dictionaries

Drawbacks:
* Runs on Linux systems only
* May need some work to adapt it for Eiffel

=== KTranslator ===
This tool helps the translation of single words, independently from the used application.

Advantages:
* Application independant, used while running the program

Drawbacks:
* Uses popups to display the translation of the word
* It works exclusively with dictionaries
* It isn't really a tool that can do big work of translation

[http://ktranslator.sourceforge.net/index.html More]

=== Other tools to watch ===
* xgettext
* poxml (Definition: tools for using PO-files to translate DocBook XML files This is a collection of tools that facilitate translating DocBook XML files using gettext message files (PO-files).)
* [http://po4a.alioth.debian.org/ po4a] (The po4a (po for anything) project goal is to ease translations (and more interestingly, the maintenance of translations) using gettext tools on areas where they were not expected like documentation.)
* xml2po
* [http://nlso-objects.sourceforge.net/ NLSO] (for Windows)
* Kartouche

=== Conclusions ===

== Writing our own tools ==
That could be a good idea to write our own tools to do these tasks, in particular for a distributed translation (everybody can help on internet).
With this I mean a little website that allows us to do the actual translation directly via the web, and it should automatically create the new version of the language file to be downloaded from all the users.

Another thing that this tool can do for us is scanning the entire source code looking for new translations to be done, and propose them to the users... can be interesting because we than only have to click on a link and all the work (not the translations) is done for us.

Advantages
* independent from the architecture
* independent from the language file format
Drawbacks
* can be a lot of work

= Problems, Ideas, Further Development =
some problems/ideas that may go with the other parts of this project as well:
* The i18n can be put online as a database that everyone can update (thus helping translation in many more languages)
After the user has chosen a language the entries still untranslated will be displayed so that the user can insert his own version of the translated message
There may be the problem of some users who may have fun writing bad, strange, crazy, nonsense sentences, thus a solution is needed since it clearly can't be checked by the project team
* The main database can be updated as soon as something new for eiffel comes out (by updated here is meant simply adding the new messages, not their translations)
* A 2nd database can be placed on the user's pc and autoupdate Eiffel and the additional content one may have when needed (when a new version of Eiffel or a new tool is available for example), with this system one doesn't need to download every time a new file containing some information already downloaded before
* Unicode problem (maybe a collaboration with the Unicode group can help)

Internationalization/file format

2006-05-31T16:49:29Z

Etienner: added page to our category

[[Category:Projects]]
[[Category:Internationalization]]

==Summary==

Here [[Internationalization|we]] evaluate various file formats used for the translation of programs. For the moment we are considering:

* XML
* po
* [http://transolution.python-hosting.com xliff] (good description in this homepage)
* create an own format

==PO Files==

===Format of PO files===

A PO file has an entry for each string that has to be translated. There are two kind of them, a "normal" one and one that involves plural forms.

====Normal entry====

Here is the general structure of a "normal" entry:

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string
msgstr translated-string

Where the ''translator-comments'' are created and maintained exclusively by the translator, this comments have some white space immediately following the #. The other comments are created by the program that created the PO file.
After the special comment "#," there can be some ''flags'', as ''fuzzy'' shows that the msgstr string might not be a correct translation, i.e. the translator is not sure of his work.
The 'untranslated-string' is the untranslated string as it appears in the original program source. The ''translated-string'' is (as the name suggests) the translated string, if there is no translation it is an empty string.

====Plural form entry====

white-space
# translator-comments
#. automatic-comments
#: reference...
#, flag...
msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

===Supported character encodings===

character encodings that can be used are limited to those supported by both GNU libc and GNU libiconv. These are: ASCII, ISO-8859-1, ISO-8859-2, ISO-8859-3, ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, ISO-8859-9, ISO-8859-13, ISO-8859-15, KOI8-R, KOI8-U, CP850, CP866, CP874, CP932, CP949, CP950, CP1250, CP1251, CP1252, CP1253, CP1254, CP1255, CP1256, CP1257, GB2312, EUC-JP, EUC-KR, EUC-TW, BIG5, BIG5-HKSCS, GBK, GB18030, SHIFT_JIS, JOHAB, TIS-620, VISCII, UTF-8.

I think they are a lot...

===Po Editors===

* poEdit
* KBabel
* Gtranslator
* LocFactoryEditor (XLIFF and PO editor for Mac OSX)

===Positive aspects===

* Powerful plural handling
* Format created for translation purpose
* Easy for humans to read
* Used by gettext, kbabel, rosetta and many other programs
* Support and elaboration tools for almost all plattforms

===Negative aspects===

* We have to write a PO parser

==XML==

===Format of XML===

XML is used for example by Trolltech for their .ts files. Here an example of .ts file, generated my lupdate (a tool made by trolltech that extracts translatable text from the C++ source code of the Qt application, see [[Internationalization/tool evaluation|here]] for further information):

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation type="unfinished"></translation>
</message>
</context>
</TS>

And after the translation (for example with Qt Linguist) it would look like this:

<!DOCTYPE TS><TS>
<context>
<name>MyExample</name>
<message>
<source>i18n=Internationalization</source>
<translation>i20e=Internazionalizzazione</translation>
</message>
</context>
</TS>

===Positive aspects===

* full support for unicode character encodings
* There is already a parser in the EiffelBase
* In trolltech's opinion it's a human readable text

===Negative aspects===

* Not everybody knows it
* Microsoft seeks XML-related patents that could restrict the use of XML (there should be a "Very negative aspect" section)
:Thank you for that link, but the Microsoft Patents apply only on the Office dialect of XML, which it uses to save its own documents... they DON'T apply to the XML standard and general use (as we would create our format). [[User:Trosim|Trosim]]
* In my opinion it's not a human readable text (Fortunately not all human beings are Computer scientists)

==New Format==

===Format of our Format===

* It doesn't exist yet, so we don't know how it looks like.
* We could give our own extension to the file format for example .et (eiffel translation) or .babe (babylon eiffel) or .eint (eiffel i18n) ... (''huge'' advantage)

===Positive aspects===

* Free to do what we want
* We don't have to care about licenses
* Possibility to make it the best human readable format
* So that we can say that we invented a new file format
* Better integrated and consistent with eiffel syntax

===Negative aspects===

* A new format? Why should we be different?
* Do more work as needed (there are already good formats)
* Long time until it becomes famous

==Conclusions==

Our decision is to use the po file format.

==References==

* [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html Gettext manual (for PO files)]
* [http://librarian.launchpad.net/2395419/it.po Example of a PO file (from rosetta)]
* [http://news.com.com/2100-1013_3-5146581.html Microsoft and XML]
* [http://en.wikipedia.org/wiki/XML#Quick_syntax_tour XML syntax description on Wikipedia]
* [http://doc.trolltech.com/4.1/i18n.html Trolltech i18n]
* [http://translate.sourceforge.net/wiki/ open source i18n and l10n project]

Internationalization/feasibility

2006-05-31T16:48:43Z

Etienner: added page to our category

[[Category:Internationalization]]

==Summary==
Our task is to find out the possible main difficulties that will be encoutered during both design and implementation of the [[internationalization|i18n]] framework.

Main focuses are:
* state of the Unicode implementation
* strings usage in Vision2
* runtime generated/composed strings with variables

==String types and their possible difficulties==

===Constant String===
Workbench_name: STRING is "EiffelStudio"

====Difficulties====
====Possible Solution====

===String with label specific syntax===
m_to_lower: STRING is "Set to &Lowercase%TCtrl+Shift+U"

====Difficulties====
* Bad readability for the traducer
* Shortcuts conflict
====Possible Solution====

===Composed String===
m_About:STRING is
once
Result:"&About"+ Workbench_name + "..."
end

====Difficulties====
====Possible Solution====

===Runtime composed String containging queries===
w_Project_directory_not_exist (file_name, dir_name: STRING):STRING is
do
create Result.make(256)
Result.append ("%NCannot open project '")
Result.append (file_name)
Result.append ("'.%N%NMake sure you have a complete EIFGEN directory in '")
Result.append (dir_name)
Result.append ("'.")
end

====Difficulties====
====Possible Solution====
==Important Classes==

====Defining and formatting Strings====

* INTERFACE_NAMES: Include all the strings of the ES interface (buttons, labels, shortcuts...)
* TEXT_FORMATTER: Text formatter for all text output formatting
* ERROR: Error object sent by the compiler to the workbench. Ancestor class of all error classes (WARNING, SYNTAX_ERROR, INTERRUPT_ERROR, LACE_ERROR,EIFFEL_ERROR...)

====Implementations====

* STRING_GENERAL: "Common ancestors to all STRING classes." [base]
** STRING: [base]
*** UC_STRING: Unicode strings [gobo]
**** UC_UTF8_STRING: Unicode strings with UTF-8 encoding
** STRING_32: [base]
* UC_CHARACTER: Unicode characters [gobo]

* KS_STRING: Portable interface for class STRING [Gobo]
* cluster: "gobo.kernel.unicode"

* WIDE_CHARACTER: Unicode characters, with comparison operations [base]

Category:Internationalization

2006-05-31T16:46:50Z

Etienner:

List of pages related to the [[internationalization]] project

Internationalization

2006-05-31T16:45:13Z

Etienner: created category of Internationalization project

[[Category:Projects]]
[[Category:Internationalization]]
[[Image:ebabylon.png|right|frame| Our Eiffel Tower of Babylon]]

=Overview=

''"Many [people] would simply love seeing their computer screen showing a lot less of English, and far more of their own language."'' -- gettext doc

Our aim is not only to provide a framework to ease the translation of Eiffel-written applications, allowing the user to chose his/her preferred language at runtime, but also to let the developer access information and formats based on users' locale.

==What is internationalisation?==

The first thing that comes to mind is translation. But internationalisation isn't restricted to enabling translation: it includes making it possible to localise notations (time, date, numbers), measures, paper size, and much more.

==What should we achieve?==
*Applications should be able to load localized strings at runtime and be provided with localized format strings (e.g date format).
*Developers can use tools that automagically extract strings from source code and can try to get them translated in a file to distribute along with the application.
*Users will still be unhappy and get depressed ''but in their own language'', which we can all agree is a significant step forward.

==Who should do the achieving?==
It is clear that our [[User:Carlo|Glorious Leader]] should be credited with any achieving. However he will have to delegate things somewhat, as he is is not [http://en.wikipedia.org/wiki/Aleksei_Grigorievich_Stakhanov Stakhanov].
We can probably divide the project into the following parts, modulo preliminary research:

''This should be probably be completed after we have got past M1''

=Milestones=

==M2: May 5 ==
* [[Internationalization/feasibility|feasibility]]: look at string classes (unicode and not) and how strings are used in EiffelStudio (creation, composition) ''(Ivano, Carlo, Leo, Hong)''
* [[Internationalization/file_format|file format]]: compare existing file formats for dictionaries ''(Etienne, Andreas)''
* [[Internationalization/tool_evaluation|tool evaluation]]: list and compare existing translation tools ''(Christian, Martino)''

==M3: June ??? ==
* write an initial .po to start translating and have a .mo for testing ''(Leo, Carlo)''
* [[Internationalization/mo parser|mo parser]]: extract translated stings from .mo files ''([[User:etienner|Etienne]], [[User:Trosim|Martino]])''
* [[Internationalization/code_parser|code parser]]: extract strings to be translated from source code and generate .po file
* [[Internationalization/translation function|translation function]]: map hard-coded strings to translated strings
** find a solution with templates
** globality: how to implement, the object should be shared between all modules (see [[Internationalization/class_structure|class structure]])
* collaborate with the [[Vision2_and_Unicode]] team
* Test unicode support in Vision2 ''(Ivano)''
* compile a [[Internationalization/features list|list of basic features]] to provide (e.g. date/time format, system locale) ''[deferred]''

= Possible future developments =

* collaborate with the [[ESWizard]] team. Create wizards for programms with translation facilities.

=Relevant Links=
[http://www.debian.org/doc/manuals/intro-i18n/ Introduction to i18n]

==What other people have done==
[http://doc.trolltech.com/4.1/i18n.html internationalisation with QT]

[http://oss.erdfunkstelle.de/kde-i18n/tiki-index.php?page=miniHowtoGui howto for internationalisation of KDE programs ]

[http://l10n.kde.org/docs/translation-howto/ another KDE howto (doesn't Gnome do any internationalisation?)]

=Team=

Everyone interested in this project is welcome to [http://origo.ethz.ch/cgi-bin/mailman/listinfo/es-i18n join our mailinglist] es-i18n@origo.ethz.ch

* Project Leader: [[User:Carlo|Carlo Vanini]]
* [[User:leo| Leo Fellmann]]
* [[User:kiwi| Ivano Somaini]]
* [[User:murbi|Andreas Murbach]]
* [[User:etienner|Etienne Reichenbach]]
* [[User:hong |Hong Zhang]]
* [[User:cconti | Christian Conti]]
* [[User:Trosim |Martino Trosi]]
* [[User:Schoelle| Bernd Schoeller]]

Internationalization/plural forms

2006-05-31T15:56:05Z

Etienner: /* =Three forms, special case for numbers ending in 1[2-9] */

The information about the plural form selection has to be stored in the header entry of the PO file (the one with the empty msgid string). The plural form information looks like this:
::Plural-Forms: nplurals=2; plural=n == 1 ? 0 : 1;

The ''nplurals'' value must be a decimal number which specifies how many different plural forms exist for this language. The string following plural is an expression which is using the C language syntax. Exceptions are that no negative numbers are allowed, numbers must be decimal, and the only variable allowed is ''n''. This expression will be evaluated whenever one of the [[internationalization/translation_function|translation function]] for a "plural form" is called. The numeric value passed to these functions is then substituted for all uses of the variable n in the expression. The resulting value then must be greater or equal to zero and smaller than the value given as the value of ''nplurals''.

==Rules==
The (at this point) known rules are:
===Only one form===

Some languages only require one single form. There is no distinction between the singular and plural form. An appropriate header entry would look like this:

:Plural-Forms: nplurals=1; plural=0;

:Languages with this property include:
:*Finno-Ugric family
:*Hungarian
:*Asian family
:*Japanese, Korean
:*Turkic/Altaic family
:*Turkish

===Two forms, singular used for one only===

This is the form used in most existing programs since it is what English is using. A header entry would look like this:

:Plural-Forms: nplurals=2; plural=n != 1;
:::(Note: this uses the feature of C expressions that boolean expressions have to value zero or one.)

:Languages with this property include:
:*Germanic family
:*Danish, Dutch, English, German, Norwegian, Swedish
:*Finno-Ugric family
:*Estonian, Finnish
:*Latin/Greek family
:*Greek
:*Semitic family
:*Hebrew
:*Romanic family
:*Italian, Portuguese, Spanish
:*Artificial
:*Esperanto

===Two forms, singular used for zero and one===

Exceptional case in the language family. The header entry would be:

:Plural-Forms: nplurals=2; plural=n>1;

:Languages with this property include:
:*Romanic family
:*French, Brazilian Portuguese

===Three forms, special case for zero===

The header entry would be:

:Plural-Forms: nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n != 0 ? 1 : 2;

:Languages with this property include:
:*Baltic family
:*Latvian

===Three forms, special cases for one and two===
The header entry would be:

:Plural-Forms: nplurals=3; plural=n==1 ? 0 : n==2 ? 1 : 2;

:Languages with this property include:
:*Celtic
:*Gaeilge (Irish)

===Three forms, special case for numbers ending in 1[2-9]===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n%10==1 && n%100!=11 ? 0 : \
::::::::n%10>=2 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Baltic family
:*Lithuanian

===Three forms, special cases for numbers ending in 1 and 2, 3, 4, except those ending in 1[1-4]===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n%10==1 && n%100!=11 ? 0 : \
::::n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Slavic family
:*Croatian, Czech, Russian, Slovak, Ukrainian

===Three forms, special case for one and some numbers ending in 2, 3, or 4===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n==1 ? 0 : \
::::n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Slavic family
:*Polish

===Four forms, special case for one and all numbers ending in 02, 03, or 04===

The header entry would look like this:

:Plural-Forms: nplurals=4; \
::plural=n%100==1 ? 0 : n%100==2 ? 1 : n%100==3 || n%100==4 ? 2 : 3;

:Languages with this property include:
:*Slavic family
:*Slovenian

Internationalization/translation function

2006-05-31T14:47:55Z

Etienner: plural forms link

==Introduction==

The problem for the translation is that we need a way to translate tree different types of string:

* normal strings
<code>[eiffel,N] "A normal string"</code>
* composed string
<code>[eiffel,N] "A string composed by "+count.out+" strings, like this or "+a_string+"%N"</code>
* strings with plural form
<code>[eiffel,N]
if n=1 then
Result := "a string"
else
Result := "strings"
end
</code>

An other problem is that not all languages have the same rule for plural forms, see below and [[internationalization/plural_forms|here]] for more details.

==Possible solution==

I [[Talk:Internationalization/translation_function|'''propose''']] a solution, with four functions (I've got little fantasy for the function names):

<code>[eiffel,N]
i18n (a_string : STRING) : STRING
--Function for the translation of normal strings

i18n_pl (strings : TUPLE[STRING]; form : INTEGER) : STRING
--Function for the translation of normal strings
-- with plural form

i18n_comp (a_string : STRING; args : TUPLE) : STRING
--Function for the translation of composit strings

i18n_comp_pl (strings : TUPLE[STRING]; args : TUPLE; form : INTEGER) : STRING
--Function for the translation of composit
-- strings with plural forms
</code>

With this solution, the programmer should write composit strings as (or in a similar way) he would do it for the C ''printf'' function. The composit string above would be somthing like this:

<code>[eiffel,N] i18n_comp("A string composed by %a1 strings, like this or %a2%N", [count, a_string])
</code>

where ''%a1'' is the place where the first string in the argument tuple (here ''count'') has to go. I've [[Talk:Internationalization/translation_function|'''choosen''']] that the items in the argument tuple are of type ANY, to insert them in the string we have to apply the functon ''out'' to all items.

===An example===

A complete example of how it could work.

----
Piece of code without internationalization:

<code>[eiffel,N]

[...]
if n = 1
io.put_string("Remove file "+file_name.out+"?%N")
else
io.put_string ("Delete following files?"+list.out+" there are "
+list.count.out+"files%N")
end
[...]

</code>

----

Piece of code with internationalization:

<code>[eiffel,N]

[...]
io.put_string(["Remove file %a1?%N",
"Delete following files? %a2 there are %a3 files%N"],
[file_name, list, list_count], n)
[...]
</code>

the content of the PO file would be:

#: A comment
msgid "Remove file %a1?%N"
msgid_plural "Delete following files? %a2 there are %a3 files%N"
msgstr[0] "Rimuovere il file %a1?"
msgstr[1] "Cancellare i seguenti file? %a2 ce ne sono %a3%N"

You might ask yourself, ''why a function for all the pluralforms''?

The number of plural forms differ between languages. This is somewhat surprising for those who only have experiences with Romanic and Germanic languages since here the number is the same (there are two).
But other language families have only one form or many forms. For example,

In Polish the translation of file is plik, and the plural forms are:

:2,3,4 pliki
:5-21 pliko'w
:22-24 pliki
:25-31 pliko'w
:and so on...

for more information about this topic go [http://www.gnu.org/software/gettext/manual/html_mono/gettext.html#SEC150 here]

==Additional things==

A nice thing whe could do, when parsing the source code of a program to internationalize, is to recognize comments about the string.
For example:

<code>[eiffel,N]

[...]
io.put_string(["Remove file %a1?%N",
"Delete following files? %a2 there are %a3 files%N"],
[file_name, list, list_count], n)
--A comment of the author
[...]
</code>

woud then look like this in the PO file:

#. A comment of the author
#. Function name, class name, file path
#. %a1 = file_name
#. %a2 = list
#. %a3 = list_count
msgid "Remove file %a1?%N"
msgid_plural "Delete following files? %a2 there are %a3 files%N"
msgstr[0] "Rimuovere il file %a1?"
msgstr[1] "Cancellare i seguenti file? %a2 ce ne sono %a3%N"

Internationalization/plural forms

2006-05-31T14:44:27Z

Etienner: list of known forms

The information about the plural form selection has to be stored in the header entry of the PO file (the one with the empty msgid string). The plural form information looks like this:
::Plural-Forms: nplurals=2; plural=n == 1 ? 0 : 1;

The ''nplurals'' value must be a decimal number which specifies how many different plural forms exist for this language. The string following plural is an expression which is using the C language syntax. Exceptions are that no negative numbers are allowed, numbers must be decimal, and the only variable allowed is ''n''. This expression will be evaluated whenever one of the [[internationalization/translation_function|translation function]] for a "plural form" is called. The numeric value passed to these functions is then substituted for all uses of the variable n in the expression. The resulting value then must be greater or equal to zero and smaller than the value given as the value of ''nplurals''.

==Rules==
The (at this point) known rules are:
===Only one form===

Some languages only require one single form. There is no distinction between the singular and plural form. An appropriate header entry would look like this:

:Plural-Forms: nplurals=1; plural=0;

:Languages with this property include:
:*Finno-Ugric family
:*Hungarian
:*Asian family
:*Japanese, Korean
:*Turkic/Altaic family
:*Turkish

===Two forms, singular used for one only===

This is the form used in most existing programs since it is what English is using. A header entry would look like this:

:Plural-Forms: nplurals=2; plural=n != 1;
:::(Note: this uses the feature of C expressions that boolean expressions have to value zero or one.)

:Languages with this property include:
:*Germanic family
:*Danish, Dutch, English, German, Norwegian, Swedish
:*Finno-Ugric family
:*Estonian, Finnish
:*Latin/Greek family
:*Greek
:*Semitic family
:*Hebrew
:*Romanic family
:*Italian, Portuguese, Spanish
:*Artificial
:*Esperanto

===Two forms, singular used for zero and one===

Exceptional case in the language family. The header entry would be:

:Plural-Forms: nplurals=2; plural=n>1;

:Languages with this property include:
:*Romanic family
:*French, Brazilian Portuguese

===Three forms, special case for zero===

The header entry would be:

:Plural-Forms: nplurals=3; plural=n%10==1 && n%100!=11 ? 0 : n != 0 ? 1 : 2;

:Languages with this property include:
:*Baltic family
:*Latvian

===Three forms, special cases for one and two===
The header entry would be:

:Plural-Forms: nplurals=3; plural=n==1 ? 0 : n==2 ? 1 : 2;

:Languages with this property include:
:*Celtic
:*Gaeilge (Irish)

===Three forms, special case for numbers ending in 1[2-9]==

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n%10==1 && n%100!=11 ? 0 : \
::::::::n%10>=2 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Baltic family
:*Lithuanian

===Three forms, special cases for numbers ending in 1 and 2, 3, 4, except those ending in 1[1-4]===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n%10==1 && n%100!=11 ? 0 : \
::::n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Slavic family
:*Croatian, Czech, Russian, Slovak, Ukrainian

===Three forms, special case for one and some numbers ending in 2, 3, or 4===

The header entry would look like this:

:Plural-Forms: nplurals=3; \
::plural=n==1 ? 0 : \
::::n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2;

:Languages with this property include:
:*Slavic family
:*Polish

===Four forms, special case for one and all numbers ending in 02, 03, or 04===

The header entry would look like this:

:Plural-Forms: nplurals=4; \
::plural=n%100==1 ? 0 : n%100==2 ? 1 : n%100==3 || n%100==4 ? 2 : 3;

:Languages with this property include:
:*Slavic family
:*Slovenian