Showing posts with label i18n. Show all posts
Showing posts with label i18n. Show all posts

18 December 2015

Localization vs. Internationalization (i18n) in SQL and .NET

Internationalization is often written i18n, where 18 is the number of letters between i and n in the English word. Internationalization typically entails: Designing and developing in a way that removes barriers to localization or international deployment.

More about the term is described elegantly at w3
http://www.w3.org/International/questions/qa-i18n

----

There are many issues in handling time in systems that span globally.
One area is the database level and the other is the code level.

Prior to the introduction of the DateTimeOffset type the types offered did not let you save both the UTC time and the actual local timezone in one value. This would typically be handled by composing time upon two stored values, the datetime in UTC and the TimeZoneOffset in another field.

These shortcomings have been handled with the introduction of the DateTimeOffset value in both SQL and .NET. There is still room for improvement because we do not have the Daylight Saving Time Parameter in the type, but this is a value that is set nationally and just a pain to handle ;)

What have the larger systems been doing prior to this?

E.g.The Sitecore platform has saved all internal datetime values to the database as UTC time and let the presentation layer handle the formatting to localized time. For internal values like a publishing pipeline or an event queue having a single point of time reference is critical. This is still a relevant for system values where you want to be precise in the time comparison.

DateTimeOffset (from SQL server 2008)

http://blogs.msdn.com/b/bartd/archive/2009/03/31/the-death-of-datetime.aspx

SQL Server 2008 added a new data type named “datetimeoffset”. This is similar to the old datetime data type, with the following significant differences:
  • Internally, the time is stored in unambiguous UTC format
  • The local time zone offset is stored along with the UTC time, which allows the time to be displayed as a local time value (or converted to any another time zone offset)
  • The data type is capable of storing more precise times than datetime

DateTimeOffset (from .NET Framework 3.5)

The same type has been introduced in .NET v3.5, it can map the new SQL type.
https://msdn.microsoft.com/en-us/library/system.datetimeoffset%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396

Choosing Between DateTime, DateTimeOffset, TimeSpan, and TimeZoneInfo
https://msdn.microsoft.com/en-us/library/bb384267%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396

The Daylight Saving Time - the missing link... 

I have found one excellent resource for handling Daylight Saving Time. The idea is to let the user select there country of origin, and save the Daylight Offset value at the same time, it can also save the gmt offset if you choose.

The right way is to find the value through the .NET framework with the TimeZone information for a region. But allowing it to be part of a custom value set by the user selection can help give a simplicity around the value. 



16 June 2015

dotNET Internationalization (i18n) using po files and GNU tools



Originally posted on http://issues.umbraco.org/issue/U4-6698

Localization based on GNU Gettext strategy instead of developer driven xml files

http://www.gnu.org/software/gettext/
http://en.wikipedia.org/wiki/Gettext

http://www.gnu.org/software/gettext/manual/gettext.html#Why
Since this is a long rant, I’ll help you a bit along, the resume is -> http://poedit.net/wordpress <- so Umbraco should adopt this too to get in on this translation eco system…

Branch: https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/
The interesting part is build/Translation.cmd and the resulting pot file at
https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/src/Umbraco.Web.UI/Umbraco/config/lang/messages.pot

----

This was an open space topic presented at Codegarden 15 - I would like the Umbraco core project to refactor the internal text translation eco system and adopt GNU Gettext and po+pot files translation strategy. The purpose is to remove the developer pain that exists in working with translating strings to internal gui and open up for a larger set of tools a communities maybe on a package level as well as on umbraco core.

----

Translation, i18n or localization in asp.net using resource files is broke, many fine blog posts argument to why, and is unusable in an open source space.
http://www.expatsoftware.com/articles/2010/03/why-internationalization-is-hopelessly.html
http://manas.com.ar/blog/2009/10/01/using-gnu-gettext-for-i18n-in-c-and-asp-net.html

The current implementation in Umbraco uses a simple and clean custom xml file based system with id references to text arrays. A developer must create the string as id and the initial translation en e.g. english. The system is easily expandable to package creators and requires “specialized” translation handling. The missing “id” could be injected at runtime, but if this is an edge case error message it might never find its way to the main translation file xml. The biggest problem right now i believe is that the package eco system is running there own translation subsystems on each package leaving the Umbraco Translations fragmented and an tedious task to handle for developers.

The GNU Project has another strategy that has been in use since 1995.
https://en.wikipedia.org/wiki/Gettext/
https://www.gnu.org/software/gettext/
https://www.gnu.org/software/gettext/manual/html_node/index.html
https://docs.python.org/2/library/gettext.html#gettext its big in python and
https://developer.mozilla.org/en-US/docs/gettext and big in php

The Gettext strategy is to scan the source files and generating a pot (po template) file containing the message “id”, default keywords for localized strings are _(“”) or gettext(“”). And merge/update the po-template into the localized po files updating them so e.g. outdated translation messages are commented out.

The source files can be any code files, C# and JavaScript are supported. Comments can be entered above your code piece to get context help into the notes of the po files, and the po files also contains reference comments to where in the source the translation is in use.
https://www.gnu.org/software/gettext/manual/html_node/xgettext-Invocation.html#xgettext-Invocation

Notice the translation mechanics in base gettext are looking at plural forms also
https://github.com/neris/NGettext/blob/master/src/NGettext.Tests/BaseCatalogTest.cs

There is a large echo system build around the translation process of these file formats and tools can be used to cooperate in a global scale. http://poedit.net/ is one of the main tools and difference implementations can be found in linux and mac systems.

One of the main arguments to why Umbraco should change the strategy is this: http://poedit.net/wordpress

How could we work with this.

I have tried this out on my pet project https://github.com/janhebnes/startlist.club and I am using https://github.com/vslavik/gettext-tools-windows for the main “ngettext” scanner and running a batch for updated the translation files https://github.com/janhebnes/startlist.club/blob/master/Translation.cmd

For working with the _ and gettext in asp.net i based my implementation on http://www.fairtutor.com/fairlylocal and tweaked it a bit to introduce LocalizedDisplayNames on models and have _ on the PageView. Everything available for review here: https://github.com/janhebnes/startlist.club/tree/master/FlightJournal.Web/Translations

I have looked at other options on the po parsing side:
https://github.com/neris/NGettext or https://github.com/fsateler/gettext-cs-utils (using a tt file in the project instead of a batch)

http://www.fairtutor.com/fairlylocal uses a project build step, i have currently chosen a separate batch.

----

Disclaimer: I have not analysed the core Umbraco before so this is based on a short review of the core. (this is where your feedback is much required)
----

How do we go about and handle the change.


C# sources

Most methods are based in \src\umbraco.businesslogic\ui.cs, methods like GetText and Text and a TextService.Localize. So maybe someone has allready had thought towards GNU when designing this. (nobody came out about it at codegarden 15’) All reference the TextService.Localize. The main issue is to get the “messageid” to be detectable by code scanning up front. Changing TextService.Localize to use po sources is the easy part.

Umbraco.ui contains:
GetText (used 65 times)
Text (used 53+14+378+20+134+345+10+2120+36) 3110 times
Any use of e.g. non message ids a unusable to the scanner e.g. ui.Text(action.Alias, u) \src\umbraco.cms\businesslogic\workflow\Notification.cs and should be made detectable in some other way to get into the pot file.

Javascript sources

The translatable strings aka “messages” in the backoffice AngularJS javascript must be made scannable with the default gettext utility. Scanning Umbraco.Web.UI.Client and the js. 136 of the “messages” are in a pattern of <localize key="or">, this should be changed to either _ or gettext (see also 5.1.6 for further inspiration https://www.gnu.org/software/gettext/manual/html_node/xgettext-Invocation.html#xgettext-Invocation)
Localize is used 136 times

Once this is tweeked a new javascript method actually handling _ or gettext must be written and last the source of the translation must be rewritten to use a po or mo file as source replacing \src\Umbraco.Web\umbraco.presentation\umbraco\js\language.aspx.cs

Changing TextService.Localize to use po or mo sources

Using e.g. https://github.com/neris/NGettext or http://www.fairtutor.com/fairlylocal in the form of e.g. https://github.com/janhebnes/startlist.club/tree/master/FlightJournal.Web/Translations could be one way.


Branch Created for demonstration

https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/
Batch file added https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/build/Translation.cmd
Resulting pot based on current source https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/src/Umbraco.Web.UI/Umbraco/config/lang/messages.pot

messageid as english text

For removing a step for the developers and making translation process simplere the messageid or key must be changed to the actual english translation instead of a system key. But this can be a later step.

Potential roadmap

The core umbraco translation can be maaintained by teams on crowdin.
The package system can hook up on the same format and network, an when installed drop package po files in the lang folders, using the same system for handling strings in javascript and cs, and the same setup with pot templates per package. Depending on strategy the translating of packages could be a central service in the core. So a package manager just handles creating the pot and the eco system around translation updates and maintain po package files based on the package pot.

Feedback

I would like to get some community feedback on this feature.