Making Umbraco or any .Net CMS respect default documents

HttpHandler that helps respect the Request flow and serves the default documents of a directory instead of allowing e.g. Umbraco to kidnap the request for there purposes.

IIS does not serve the Default Document when set in Integrated pipeline mode. E.g. after installing Umbraco the httpModules will catch any traffic to the site and will return a 404 page instead of actually passing to the existing file system. In this case I wanted to add umbraco on top of an old asp and html based site. We did not want to move large amount of archive content, so using this modules allows for the classic Default Documents to be respected even if you are in a modern setup where Default Documents are being ignored. So first we check the file system for a response document if non is found the request is allowed to continue to Umbraco, allowing us to extend the existing content with Umbraco instead of having to nuke everything.

umbraco set date field value from code

Had problems with umbraco not setting date field value from code, when you reloaded the interface.

My conversion code was:

var content = contentService.GetById(3150);
//content.SetValue("publicationDate", dateValue);
content.Properties["publicationDate"].Value = dateValue;
contentService.SaveAndPublishWithStatus(content, raiseEvents: true);

Notice both formats for entering values is usable. 

The resulting data was that the value was being written to the database in the date field

SELECT TOP 1000 *

  FROM [Umbraco].[dbo].[cmsPropertyData] where contentNodeId like '3150'

When dealing with a date field that has a prior value you can see the changed value right away when refreshing the browser, when working with an empty date you have to recycle the apppool for the date to come up in the interface. 

Umbraco to Azure / Transformed Web.config MSDeploy error

Could not open Source file: Could not find a part of the path

A rather simple umbraco nuget based install and base Publishing profile from Azure getting me these types of errors:

Transformed Web.config using D:\Solutions\xxx\src\xxx.Web\Web.Release.config into obj\Release\TransformWebConfig\transformed\Web.config.
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\Web\Microsoft.Web.Publishing.targets(2294,5): Error : Could not open Source file: Could not find a part of the path 'D:\Solutions\xxx\src\xxx.Web\umbraco\Install\Views\Web.config;\umbraco\Install\Views\Web.config'.

Others having the same issue.
http://forums.iis.net/t/1217423.aspx?MSDeploy+error+Could+not+open+Source+file+Could+not+find+a+part+of+the+path+

Issue

Our default projects have a .wpp.Targets file that handles packages for msdeploy on our central build server, the package template for umbraco contains all files related to umbraco leaving "no one behind" strategy. 

wpp.Targets defines default packages information that is set throughout any publishing profiles.

Combined with the default publishing profile where a connection string wants to replace content in web.config this gives issues on web.config files other places than the root of the site. 

Our wpp.Targets contains a simple content inclusion, that makes sure the msdeploy package contains all required umbraco files even tho they are not part of the VS Project.

<ItemGroup>
  <!-- Umbraco -->
  <Content Include="css\**\*" />
  <Content Include="content\**\*" />
  <Content Include="macroscripts\**\*" />
  <Content Include="masterpages\**\*" />
  <Content Include="properties\**\*" />
  <Content Include="umbraco\**\*" />
  <Content Include="umbraco_client\**\*" />
    
  <Content Include="App_Browsers\**\*" />
  <Content Include="App_Plugins\**\*" />
  <Content Include="App_Start\**\*" />
 </ItemGroup>

http://www.asp.net/mvc/overview/deployment/visual-studio-web-deployment/deploying-to-production
A .pubxml file contains the settings that pertain to a specific publish profile. If you want to configure settings that apply to all profiles, you can create a .wpp.targets file. The build process imports these files into the .csproj or .vbproj project file, so most settings that you can configure in the project file can be configured in these files. For more information about .pubxml files and .wpp.targets files, seeHow to: Edit Deployment Settings in Publish Profile (.pubxml) Files and the .wpp.targets File in Visual Studio Web Projects.

Solution: AutoParameterizationWebConfigConnectionStrings

Several places online the proposed solution was AutoParameterizationWebConfigConnectionStrings=false as a parameter on msbuild but using VS Publishinging it was not clear how to introduce this parameter.

It turns out that you just through it in the "PropertyGroup" of the publishing profile. 
<AutoParameterizationWebConfigConnectionStrings>False</AutoParameterizationWebConfigConnectionStrings>

This is ninja kungfu knowledge that is apparently only learned after several hours of fidling about... any msbuild property can be set on the "PropertyGroup" of a profile.. enjoy.

The only change left is that the release profile must replace the connection string to the production connection string is otherwise would be set on the publishing replace event.

Double bonus solution

The PropertyGroup> <AutoParameterizationWebConfigConnectionStrings can be set in the Wpp.Targets file and thereby allowing for a "default" wpp.Targets that can handle imported profiles from Azure without requiring changes in the imported profile pubxml.

Our wpp.Targets on all Umbraco projects is now:

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 <ItemGroup>
  <!-- Umbraco -->
  <Content Include="css\**\*" />
  <Content Include="content\**\*" />
  <Content Include="macroscripts\**\*" />
  <Content Include="masterpages\**\*" />
  <Content Include="properties\**\*" />
  <Content Include="umbraco\**\*" />
  <Content Include="umbraco_client\**\*" />
    
  <Content Include="App_Browsers\**\*" />
  <Content Include="App_Plugins\**\*" />
  <Content Include="App_Start\**\*" />

 </ItemGroup>
 <PropertyGroup>
  <AutoParameterizationWebConfigConnectionStrings>False</AutoParameterizationWebConfigConnectionStrings>
  <AfterAddIisSettingAndFileContentsToSourceManifest Condition="'$(AfterAddIisSettingAndFileContentsToSourceManifest)'==''">
     $(AfterAddIisSettingAndFileContentsToSourceManifest);
     MakeEmptyFolders;
  </AfterAddIisSettingAndFileContentsToSourceManifest>
 </PropertyGroup>
 <Target Name="MakeEmptyFolders">
  <Message Text="Adding empty folders" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\App_Data" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\App_Code" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\css" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\imagecache" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\macroscripts" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\masterpages" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\media" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\usercontrols" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\views\partials" />
  <MakeDir Directories="$(_MSDeployDirPath_FullPath)\views\macropartials" />
 </Target>
</Project>

dotNET Internationalization (i18n) using po files and GNU tools

Originally posted on http://issues.umbraco.org/issue/U4-6698

Localization based on GNU Gettext strategy instead of developer driven xml files

http://www.gnu.org/software/gettext/
http://en.wikipedia.org/wiki/Gettext

http://www.gnu.org/software/gettext/manual/gettext.html#Why
Since this is a long rant, I’ll help you a bit along, the resume is -> http://poedit.net/wordpress <- so Umbraco should adopt this too to get in on this translation eco system…

Branch: https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/
The interesting part is build/Translation.cmd and the resulting pot file at
https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/src/Umbraco.Web.UI/Umbraco/config/lang/messages.pot

----

This was an open space topic presented at Codegarden 15 - I would like the Umbraco core project to refactor the internal text translation eco system and adopt GNU Gettext and po+pot files translation strategy. The purpose is to remove the developer pain that exists in working with translating strings to internal gui and open up for a larger set of tools a communities maybe on a package level as well as on umbraco core.

----

Translation, i18n or localization in asp.net using resource files is broke, many fine blog posts argument to why, and is unusable in an open source space.
http://www.expatsoftware.com/articles/2010/03/why-internationalization-is-hopelessly.html
http://manas.com.ar/blog/2009/10/01/using-gnu-gettext-for-i18n-in-c-and-asp-net.html

The current implementation in Umbraco uses a simple and clean custom xml file based system with id references to text arrays. A developer must create the string as id and the initial translation en e.g. english. The system is easily expandable to package creators and requires “specialized” translation handling. The missing “id” could be injected at runtime, but if this is an edge case error message it might never find its way to the main translation file xml. The biggest problem right now i believe is that the package eco system is running there own translation subsystems on each package leaving the Umbraco Translations fragmented and an tedious task to handle for developers.

The GNU Project has another strategy that has been in use since 1995.
https://en.wikipedia.org/wiki/Gettext/
https://www.gnu.org/software/gettext/
https://www.gnu.org/software/gettext/manual/html_node/index.html
https://docs.python.org/2/library/gettext.html#gettext its big in python and
https://developer.mozilla.org/en-US/docs/gettext and big in php

The Gettext strategy is to scan the source files and generating a pot (po template) file containing the message “id”, default keywords for localized strings are _(“”) or gettext(“”). And merge/update the po-template into the localized po files updating them so e.g. outdated translation messages are commented out.

The source files can be any code files, C# and JavaScript are supported. Comments can be entered above your code piece to get context help into the notes of the po files, and the po files also contains reference comments to where in the source the translation is in use.
https://www.gnu.org/software/gettext/manual/html_node/xgettext-Invocation.html#xgettext-Invocation

Notice the translation mechanics in base gettext are looking at plural forms also
https://github.com/neris/NGettext/blob/master/src/NGettext.Tests/BaseCatalogTest.cs

There is a large echo system build around the translation process of these file formats and tools can be used to cooperate in a global scale. http://poedit.net/ is one of the main tools and difference implementations can be found in linux and mac systems.

One of the main arguments to why Umbraco should change the strategy is this: http://poedit.net/wordpress

How could we work with this.

I have tried this out on my pet project https://github.com/janhebnes/startlist.club and I am using https://github.com/vslavik/gettext-tools-windows for the main “ngettext” scanner and running a batch for updated the translation files https://github.com/janhebnes/startlist.club/blob/master/Translation.cmd

For working with the _ and gettext in asp.net i based my implementation on http://www.fairtutor.com/fairlylocal and tweaked it a bit to introduce LocalizedDisplayNames on models and have _ on the PageView. Everything available for review here: https://github.com/janhebnes/startlist.club/tree/master/FlightJournal.Web/Translations

I have looked at other options on the po parsing side:
https://github.com/neris/NGettext or https://github.com/fsateler/gettext-cs-utils (using a tt file in the project instead of a batch)

http://www.fairtutor.com/fairlylocal uses a project build step, i have currently chosen a separate batch.

----

Disclaimer: I have not analysed the core Umbraco before so this is based on a short review of the core. (this is where your feedback is much required)
----

How do we go about and handle the change.

C# sources

Most methods are based in \src\umbraco.businesslogic\ui.cs, methods like GetText and Text and a TextService.Localize. So maybe someone has allready had thought towards GNU when designing this. (nobody came out about it at codegarden 15’) All reference the TextService.Localize. The main issue is to get the “messageid” to be detectable by code scanning up front. Changing TextService.Localize to use po sources is the easy part.

Umbraco.ui contains:
GetText (used 65 times)
Text (used 53+14+378+20+134+345+10+2120+36) 3110 times

Any use of e.g. non message ids a unusable to the scanner e.g. ui.Text(action.Alias, u) \src\umbraco.cms\businesslogic\workflow\Notification.cs and should be made detectable in some other way to get into the pot file.

Javascript sources

The translatable strings aka “messages” in the backoffice AngularJS javascript must be made scannable with the default gettext utility. Scanning Umbraco.Web.UI.Client and the js. 136 of the “messages” are in a pattern of <localize key="or">, this should be changed to either _ or gettext (see also 5.1.6 for further inspiration https://www.gnu.org/software/gettext/manual/html_node/xgettext-Invocation.html#xgettext-Invocation)

Localize is used 136 times

Once this is tweeked a new javascript method actually handling _ or gettext must be written and last the source of the translation must be rewritten to use a po or mo file as source replacing \src\Umbraco.Web\umbraco.presentation\umbraco\js\language.aspx.cs

Changing TextService.Localize to use po or mo sources

Using e.g. https://github.com/neris/NGettext or http://www.fairtutor.com/fairlylocal in the form of e.g. https://github.com/janhebnes/startlist.club/tree/master/FlightJournal.Web/Translations could be one way.

Branch Created for demonstration

https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/
Batch file added https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/build/Translation.cmd
Resulting pot based on current source https://github.com/janhebnes/Umbraco-CMS/blob/dev-v7-localization-using-gettext/src/Umbraco.Web.UI/Umbraco/config/lang/messages.pot

messageid as english text

For removing a step for the developers and making translation process simplere the messageid or key must be changed to the actual english translation instead of a system key. But this can be a later step.

Potential roadmap

The core umbraco translation can be maaintained by teams on crowdin.
The package system can hook up on the same format and network, an when installed drop package po files in the lang folders, using the same system for handling strings in javascript and cs, and the same setup with pot templates per package. Depending on strategy the translating of packages could be a central service in the core. So a package manager just handles creating the pot and the eco system around translation updates and maintain po package files based on the package pot.

Feedback

I would like to get some community feedback on this feature.