File system places accessible through WinRT API

In the previous posts I described the restrictions in file system access dictated by Windows 8 for WinRT apps. With user’s help and by using file and folder pickers your app can get an access to any file system object, but there is a limited number of folders accessible for the app via direct API calls. Here I want to summarize information about these folders.

Local app data

Local data folder allows you to store any application-related files. The folder is private and isolated so any other Metro apps are not allowed to access the stored files.

The folder will be completely erased if the app is uninstalled.

// C#
StorageFolder localFolder = Windows.Storage.ApplicationData.Current.LocalFolder;

// XAML
<Image Source="ms-appdata:///local/myFile.png"/>

Roaming app data

Roaming folder provides you a way to store the files and synchronize them across multiple systems associated with the same Live ID (Microsoft Account).

Roaming folder size is limited and can be determined by using ApplicationData.Current.RoamingStorageQuota and ApplicationData.Current.RoamingStorageUsage variables. If folder size exceeds the quota, Windows suspends data synchronization.

// C#
StorageFolder roamingFolder = Windows.Storage.ApplicationData.Current.RoamingFolder;

// XAML
<Image Source="ms-appdata:///roaming/myFile.png"/>

Temporary app data

Temporary folder can be used to store the app’s temporary data.

Files stored in the folder can be removed at any time by system or by user via Disk Cleanup.

// C#
StorageFolder tempFolder = Windows.Storage.ApplicationData.Current.TemporaryFolder;

// XAML
<Image Source="ms-appdata:///temp/myFile.png"/>

App installed location

Metro-style apps can access all the files included in the package and installed to the target system.

// C#
StorageFolder installFolder = Windows.ApplicationModel.Package.Current.InstalledLocation;

// XAML
<Image Source="ms-appx:///myFile.png"/>

Downloads folder

User’s Downloads folder is available for your app if you need to save downloaded files.

This folder is available in write-only mode. Your app can use it to save files but it is not able to read the files downloaded by other apps.

StorageFolder tempFolder = Windows.Storage.DownloadsFolder;

Documents library

User's documents folder.

To be able to access this folder you need to specify Document Library capability in the app’s manifest. Additionally, File Type Association must be declared for the each document type that will be accessed by the app.

StorageFolder folder = Windows.Storage.KnownFolders.DocumentsLibrary;

Music library

User's music folder.

To be able to access this folder you need to specify Music Library capability in the app’s manifest.

StorageFolder folder = Windows.Storage.KnownFolders.MusicLibrary;

Pictures library

User's pictures folder.

To be able to access this folder you need to specify Pictures Library capability in the app’s manifest.

StorageFolder folder = Windows.Storage.KnownFolders.PicturesLibrary;

Videos library

User's videos folder.

To be able to access this folder you need to specify Videos Library capability in the app’s manifest.

StorageFolder folder = Windows.Storage.KnownFolders.VideosLibrary;

Removable devices

WinRT provides access to files and folder located at the removable devices such as USB sticks and external hard drives using RemovableDevices folder.

To be able to access this folder you need to specify Removable Storage capability in the app’s manifest. Additionally, File Type Association must be declared for the each file type that will be accessed by the app.

StorageFolder folder = Windows.Storage.KnownFolders.RemovableDevices;

Home group

HomeGroup folder allows you to access files and folders shared by other home computers.

Your application must to declare at least one library access capability (Pictures, Music or Videos) to be able to use this location. Shared Documents Libraries are not available for the Metro apps even if the corresponding capability is declared in the manifest.

StorageFolder folder = Windows.Storage.KnownFolders.HomeGroup;

Media Server devices

MediaServerDevices folder allows you to access files and folders shared by network media servers.

Your application must to declare at least one library access capability (Pictures, Music or Videos) to be able to use this location. Shared Documents Libraries are not available for the apps even if the corresponding capability is declared in the manifest.

StorageFolder folder = Windows.Storage.KnownFolders.MediaServerDevices;

Converting WordPress blog into statically generated site with help of Pretzel

WordPress is a good blogging engine and I was satisfied with it until a moment when I decided to have my own theme applied uniformly across the site. And here I decided to try something new, for example static site generation.

One is the most popular static site generators is Jekyll. It is a Ruby-based tool that accepts markdown text files and transforms them into HTML pages through liquid templating engine. As a result of running Jekyll on your local system you will have a fully generated site composed from static HTML pages that can be hosted in any environment. Such site does not require powerful system to be executed, can be easily migrated from host to host and is version control friendly.

If .NET environment is more familiar for you, you can try Pretzel – a C#-based project inspired by Jekyll. Pretzel is a command line utility that can prepare a site skeleton, compiles it and helps you to test the site before uploading it to the host.

First what you need to do for the new site is to prepare site structure. Pretzel supports two templating engines – liquid and Razor – and, for example, to create a site that will use Razor you need to execute

Pretzel.exe create site_name -t=razor

It will create a default site with sample data. For the default site Pretzel generates four folders and five files:

• _layouts - contains page templates
• _posts - used to keep all the blog posts
• css and img contain the site assests. While _layouts and _posts are used during site generation, these two folders will be copied to the final site. For your own site you can safely rename or delete them.
• _config.yml is a site configuration in YAML format
• about.md and index.md are the examples of non-blog pages
• atom.xml and rss.xml are used to create blog rss feed

If you will open any generated post or any page file, in the beginning of all these files you will see a similar section. This is a YAML front matter block – it is used to initialize predefined variables or to pass your custom data to the templating engine.

One of the most important variables is a layout – it defines what layout should be used to generate this page. Pretzel is trying to stay compatible with Jekyll and you can check Jekyll documentation to see what additional variables are available.

Now you can compile the site with

Pretzel.exe bake site_name

During compilation Pretzel goes through all the files in your site folder and transforms them. First all .md files are translated into HTML. After that Pretzel combines all HTML files with layouts to get the pages. And finally it copies all non-transformable files like images or CSS into the output folder. If you already tried to compile the site you will see _site folder – this is a compiled representation of your site. And now is time to test the newly created site

Pretzel.exe taste site_name -p=7878

Migration WordPress blog to Pretzel

Site generated by Pretzel is fully static and if you use standard WP commenting system you will need to switch to some third party commenting engine. It could be IntenseDebate, Disqus or Facebook plugin – it’s up to you. Switching to Disquis is easy and painful, just install WP Disqus plugin and follow the instructions and all the comments will automatically migrated.

Next you need to export your current blog posts from WordPress dashboard. Use generated file to import the posts to you new site:

Pretzel.exe import -i=wordpress -d=site_name -f=WPExport.xml

Import function does not create site – you need to create a site and import the post afterwards. After that you can create your layouts and associate them with the posts.

Tips & tricks

Partial views

Razor template engine supports partial views. You can put .cshtml or .html files in _includes folder and reuse them with help of @@Include("include_name")

URLs

Most likely you want to keep your post’s and page’s URLs. The default schema is different from the one used by WordPress but you can control it using permalink parameter in YAML front matter. For example

permalink: /blog/:year/:month/:day/:title/index.html in 2012-04-19-atdd.md

will instruct Pretzel to generate /blog/2012/04/19/atdd/index.html that is identical to post’s URL in WordPress.

Permalink parameter can be applied to any page, not only blog posts. For example to my RSS feed is linked to /blog/feed/index.html.

RSS

Pretzel generates a template for your blog feed but it should be tuned – you need to setup domain name and author name. Also Pretzel does not track posts’ publication time and if you will use RSS template as is, after the site publication your subscribers will see all previous posts in the feed. It can be avoided by adding a pub time variable to each post and by using this variable in the RSS template. Unfortunately for me, I made a mistake in time format and ended with the repopulated RSS feed.

Tags and categories

Pretzel does not generate tag clouds and any archive pages, as well as sitemap – it is your responsibility.

To display list of the categories for the post I use the following code:

@@foreach(var category in post.Categories)
{
   <a href="/blog/category/@@Filter.Slugify(category)" class="tag">@@category</a>
}

If you will check Pretzel sources (at least at a publication date) you will not see Filter.Slugify function – this change request is not yet accepted and if you’d like to see how it work please look my fork of Pretzel.

Here is the code to list all the categories

@@foreach(var category in Model.Site.Categories)
{
   <div>
      <a href="/blog/category/@@Filter.Slugify(category.Name)">@@category.Name (@@category.Posts.Count())</a>
   </div>
}

And again, Pretzel does not create categories pages for you – you need to do it manually.

Pagination

You can paginate blog index page to show, for example 10 post per page, and this behavior is also controlled in front matter:

paginate: 10
paginate_link: /blog/page/:page/index.html

This instructs Pretzel to split posts into pages by 10 and to generate index pages as /blog/page/2, /blog/page/3 and so on.

Inside of the layout you can iterate through the posts available for the current page:

@@foreach(var post in Model.Paginator.Posts)
{  
   @@Raw(post.Content)
}

For navigation all the pages can be linked together

@@if(!String.IsNullOrEmpty(@@Model.Paginator.NextPageUrl))
{
   <a href="@@Filter.PrettifyUrl(Model.Paginator.NextPageUrl)">previous posts</a>
}

So, as you can see, Pretzel is not a full featured CMS solution that can be used out-of-the box by everybody. It requires some programming skills and tuning efforts.

Pretzel is still in development and there are many things that can be added in the future but even in the current state it is stable and useful in the field.

Acceptance Test Driven Development in practice talk at TO Architecture UG

Update: Presentation slides are available on SlideShare.

Tomorrow, April 18 2012 I will speak at Toronto Architecture User Group about Acceptance Test Driven Development.

Abstract: While automated testing tools and unit testing frameworks have proven their effectiveness and are widely accepted by the agile teams practicing Test Driven Development, not many teams see a project’s test suite as the result of a joint effort between the developers, testers and domain experts. An Acceptance Test Driven Development (ATDD) approach goes one step further and brings all the stakeholders together to create an executable specification – a set of the tests written in a natural language that is understandable by the user while still being executable and verifiable. The session provides an introduction to ATDD and demonstrates the practical aspects of creating end-to-end acceptance tests, including the human side of this process. During the presentation you will also be introduced to ATDD frameworks and will learn the best practices for creating reusable and maintainable acceptance tests.

Registration at ArchitectureUG.com

My North Toronto .NET User Group Windows 8 presentation slides and samples

Samples and slides for "Getting Started with Windows 8 App Development" are available for download.

Abstract: Windows 8 represents a fundamental shift for Windows platform both in the user experience and in the development model. In this session we are going to cover the key new concepts associated with the design, development and deployment of the Windows 8 apps. We will start from an overview of Windows 8 Consumer Preview, including user experience, system architecture, execution model and the new APIs. This will be followed by a demonstration of building a Metro-style application using C# and XAML.

Deceptive simplicity of async and await

By introducing the new async/await keywords in C# 5 and by adding asynchronous APIs in .NET 4.5, Microsoft significantly lowered the bar for entering into the realm of the asynchronous programming. However, while presenting the new features of the upcoming .NET Framework release, answering the questions and helping to resolve the first issues I came to realization that this bar might be set too low.

These new additions help us to create asynchronously running methods using the familiar synchronous development model. We don’t need to segment our code to extract the parts to be executed in a separate thread or to be a callback. We don’t have to think about the threads or tasks anymore – just add await and the rest is done automatically.

That’s cool – I really like these new features and I believe they can simplify our lives, but only when used properly. Current development stack (Win7/.NET Framework 4) does not enforce the asynchronous development model. The async functionality can be introduced into existing synchronous application once the developer has fully realized the advantages of the async APIs and has mastered their proper usage.

WinRT and .NET 4.5 are different - this stack forces us to use async model. Again, this is not a bad thing but it requires some discipline and some preliminary knowledge. Right now many developers are new to the field and are experiencing the following:

Developer: “I want to use this function to read some data”
Compiler: “Good, but you may want to use await to get the string”
Developer: “OK, await is added. Can I get the string?”
Compiler: “Sure, just add async in the method header”
Developer: “Added. F5!”
Developer: “Hey, it does not work! Where is my string?”

The code looks good, it can be compiled but does not work. Why? Here are two most common mistakes I have seen so far.

Example 1

WriteFileAsync function definition:

async public void WriteFileAsync(string filename, string contents)
{
    var localFolder = Windows.Storage.ApplicationData.Current.LocalFolder;
    var file = await localFolder.CreateFileAsync(filename, 
                           Windows.Storage.CreationCollisionOption.ReplaceExisting);
    var fs = await file.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite);     
    //...
}

And this function is called as

WriteFileAsync("FileName", "Some Text");

In some cases this code will work properly. But only in some – if there is a code that depends on the file content or tries to access the same file and this code is called after WriteFileAsync(), the developer will be faced with an exception or unpredicted results.

The problem here is that the developer use awaits for the system calls but WriteFileAsync by itself is not awaitable and any code following after WriteFileAsync() call will be executed before all the expected saving operations are done. To fix the issue the method should be modified

async public Task WriteFileAsync(string filename, string contents)
{
    //...
}

And the function should be called as

await WriteFileAsync("FileName", "Some Text");

Example 2

To implement Windows 8 Sharing contract, an application should handle DataTransferManager.DataRequested event. Below you can see an example of such a handler:

async void DataRequestedHandler(DataTransferManager sender, DataRequestedEventArgs args)
{
    _fileStream = await _lastSelectedFile.OpenAsync(FileAccessMode.Read);
	
    args.Request.Data.Properties.Title = "Data Title";
    args.Request.Data.SetBitmap(_fileStream);
}

Not surprisingly, this sample does not work either. DataTransferManager does not expect that handler is asynchronous and when the await keyword returns control to the manager, the latter sees an empty data request object, assumes that the contract is not properly implemented and will not show the application on the Share panel. The fix is easy, we just need to move OpenAsync out of the method and to remove async keyword from the header, but this example demonstrates another kind of side effects – we should be very careful with the async functions when some data are expected to be returned to the external code.

These mistakes look trivial but chances are they will appear quite often due to deceptive simplicity of the new syntax. Hopefully, the next version of Visual Studio and/or ReSharper will be able to detect such situations and warn about them.

What is your opinion? Do you see a problem here?

My Metro Toronto UG presentation slides and samples

The slides and sample code from my "Windows 8 Development - Deeper Dive" presentation for Metro Toronto User Group are available for download.

1. MetroDemo demonstrates file system APIs and implementation of Share, File Picker and Search contracts.
2. TilesDemo demonstrates the basic manipulations with the application tiles.

• In addition to the contracts demonstrated during the demo, the attached code demonstrates an implementation of the Search contract.
• To see the demo in action your Pictures Library must contain at least one image tagged with some keywords. The keyword can be added in Windows Explorer by selecting the images and entering this data on the Details pane.
• What is ObservableVector.cs: ObservableCollection class is supported by Windows 8 Developer Preview but the change notification events are ignored by WinRT UI controls. ObservableVector class is a workaround to resolve the issue – it wraps an ObservableCollection object and allows us to send the notifications to the UI.

Is there a future for Microsoft Expression Blend for XAML?

I am not a fan of the visual designers and prefer to create UI by writing XAML code instead of dragging and dropping the controls. But from time to time I launch Expression Blend to create some animations, gradients or complex styles and I have to admit that Blend is a wonderful tool. Many developers like it. From the other side, Blend was created in the first place for the designers, not for the developers and from this point of view Blend is a failure. I have seen several attempts to teach UX experts to use Blend for sketching and designing and most of them failed. For the designers the learning curve is too steep, many features require some software development knowledge to use them properly.

During the BUILD conference Microsoft presented Expression Blend 5 Developer Preview that supports HTML only. Visual Studio 11 includes an improved XAML designer and some panels and controls that will look familiar for the Blend users but there is no any information on a standalone Blend for XAML. Taking into account that the whole BUILD event was designed to attract JavaScript/HTML developers and the preview status of the tools, I stayed calm waiting for the news. Until now.

Microsoft Advertising SDK for Windows 8 documentation includes an interesting table comparing the functionality available right now in a Preview Release with the features of a “Formal Launch”.

One row catches my eye:

Is it an answer for the question about the future of Blend for XAML or just an inaccuracy of the documentation?

On deploying ASP.NET MVC site as a desktop application

Sometimes you may find you in a situation when you have an ASP.NET web application and want to deploy it to a desktop PC. For example, for one of my projects I have the following requirements:

• Simple installation on any Windows-based PC. Installation and configuration should not require knowledge of the Web technologies.
• Ability to start, stop and configure the Web application from a custom desktop application.
• The Web application should be accessible for a local user and all the network users.
• In the same time it should be a regular ASP.NET application that can be deployed as a hosted solution.

It is clear that such solution will be composed from 3 main components: Web application, control application (in my case it is based on .NET/WPF), Web server. And the main question is “What should I use as a server?” To satisfy the project’s requirements, the server should have the following characteristics:

• Easily redistributable
• Configurable via API, configuration files or command line
• Should be controllable externally (Start/Stop via API, command line, etc.)
• Network support

 

CassiniDev

Already having an experience with CassiniDev server as a solution for the integration tests execution, I tried this one first. CassiniDev is an open-source project that includes a standalone Wev server with GUI and a library. The library, CassiniDev4-Lib.dll, contains CassiniDevServer class – an in-process Web server implementation.

_server = new CassiniDevServer();
_server.StartServer(“\\path\\to\\web\\content”));

Basically, that’s it - with two lines of code you have a server that hosts your ASP.NET application. At first look this is what is required:

• there is no need to install anything – the server is a part of the control application
• CassiniDevServer can be easily configured and controlled

But there are some cavities and probably the main one is performance. CassiniDev handles all the HTTP requests sequentially, in one thread. It is acceptable for testing but is not practical in production.

Another issue that kicks CassiniDev out of completion is an auto-shutdown after 60-seconds of idleness – it simply stops to handle any incoming requests.

IIS Express

IIS Express is a lightweight version of IIS. It runs as a separate process like full version of IIS web server, but this process is executed under the user’s account. IIS Express is a standalone application and can be started/stopped using a command line, unlike service-based full IIS.

Theoretically IIS Express is xcopy-deployable, but EULA states that IIS Express can only be redistributed using an official installer package. That means if you want to use IIS Express, it should be added as a prerequisite to your installer and will be installed as separated product.

IIS Express uses the same configuration mechanism as full IIS. Probably the easiest way to host the site on IIE Express, is to create own applicationhost.config, deploy it on the target system and use it to start the server:

"C:\Program Files\IIS Express\iisexpress.exe" /config:D:\Fileversum\config\applicationhost.config

Even IIS Express is installed into the system it will not be started automatically on the Windows start. You have to do it by yourself by creating some kind of launcher or incorporating this functionality into the control application, to be able to start/stop the server on-demand.

By default IIS Express does not require elevated permissions, but in the same time it default configuration does not support network requests and port 80. You need the administrative privileges to unlock these features. The post by Scott Hanselman describes the required manipulations, but in short you need to do the following:

• Configure applicationhost.config to bind your site with the public IP address.
• Allow incoming connections and add a new Windows Firewall rule using netsh utility.

Note that netsh does not exist in Windows XP and httpcfg should be used instead. That means an additional complexity for the installation script.

UltiDev Web Server Pro

UltiDev Web Server Pro is a successor for UltiDev Cassini Web Server, a freeware redistributable Web server. The product is relatively new, but it is in active development and demonstrates a good stability, performance and a reach feature set.

Like IIS Express, UWS is a standalone out-of-process Web server. It runs as a service, includes GUI frontend but also provides an API and the command line tools. UltiDev Server cannot be privately deployed, you have to install it. Fortunately the server redistributable packages for InstallShield, Advanced Installer and VS bootstrapper are available.

If you prefer to register your site during the installation, UWS provides a command line utility that can be used for this propose. If you need more control under the site’s lifecycle, a .NET-based API exposed from UWS.Configuration.dll library might be an option. Below you can see a sample code demonstrating site registration:

var app = Metabase.GetWebAppEntry(ApplicationGUID, false);
app.VirtualDirectory = "/Fileversum";
app.PhysicalDirectory = “\\path\\to\\web\\content”;
app.ApplicationName = "Fileversum";
app.AppType = ApplicationType.AspNetOrStaticHtml;
app.ListenAddresses.AddRange(new[] { new ListenAddress(80) });

Metabase.RegisterApplication(RuntimeVersion.AspNet_4, true, app);

And un-registration:

Metabase.UnregisterApplication(ApplicationGUID);

Both operations (registration and un-registration) require the administrative rights. So if this code is incorporated into the control application, it must be able to request the elevated partitions.

As soon as the site is registered, it will be available for the local and the network users. No additional actions are required to start the server or site after Windows reboot.

Additionally, to simplify the site deployment, UWS supports auto detection of the available ports.

My CTTDNUG presentation slides and samples

I published the slides and sample code from my presentation about Windows 8 at Canada's Technology Triangle .NET User Group meeting. The slides are also available on Slideshare.

Feel free to contact me if you want to discuss them.

WinRT and Windows Property System part 2: searching for files by metadata

The functions provided by WinRT to store and read files metadata are useful when you want to help your users with the files categorization. But what is even more interesting, you can use the saved properties to search for files.

For example, you can use the following code to get all the photos taken using Pentax camera and saved as .jpg files into the Pictures library:

var queryOptions = new QueryOptions(CommonFileQuery.DefaultQuery, new[] { ".jpg" });
queryOptions.FolderDepth = FolderDepth.Deep;
queryOptions.ApplicationSearchFilter = "cameramake:pentax";

StorageFileQueryResult query = 
          KnownFolders.PicturesLibrary.CreateFileQueryWithOptions(queryOptions);

IReadOnlyList files = await query.GetFilesAsync();
foreach (var file in files)
{
   Debug.WriteLine(file.FileName);
}

ApplicationSearchFilter is an Advanced Query Syntax string and is optional. If it is not defined the query will simply return all .jpg files.

The first parameter in the QueryOptions constructor defines type of the query. Using this parameter you can control the search depth and the sorting method. CommonFileQuery.DefaultQuery means shallow search – the query engine will not search for the files in the subfolders of the target folder. This behavior can be altered by assigning FolderDepth.Deep value to QueryOptions.FolderDepth property.

All the ordered queries (OrderByTitle, OrderByDate, etc.) use the deep search by default and return the flat list of all the matched files or folders in all the subfolders. But what is equally important, these queries use different methods for retrieving the results. Let’s see the following queries:

var queryOptions = new QueryOptions(CommonFileQuery.DefaultQuery, new[] { ".jpg" });
queryOptions.FolderDepth = FolderDepth.Deep;
var query = KnownFolders.PicturesLibrary.CreateFileQueryWithOptions(queryOptions);
IReadOnlyList files = await query.GetFilesAsync();

var queryOptions = new QueryOptions(CommonFileQuery.OrderByDate, new[] { ".jpg" });
var query = KnownFolders.PicturesLibrary.CreateFileQueryWithOptions(queryOptions);
IReadOnlyList files = await query.GetFilesAsync();

In theory, both queries should return the same number of the files. But when I tested this code on my system, the first query returned 24 files and the second – just 3 files.

This happened because all the ordered queries use index database to search for the files while the default query inspects the file system to find the matches. On my system the index was somehow corrupted and the query was not able to get all the files. Rebuilding the index solved the issue – both queries returned the same number of files.

Using the index database for the search helps to reduce search time and resources usage – even such a simple query was 1.5 times faster in the indexed mode - but it might be a good idea to use the default mode for the critical queries, to be sure that you received a complete view.

It is very likely that the StorageFile objects returned by the queries will be used to extract metadata and generate thumbnail for presenting to the user. As this scenario is very common, WinRT provides a class that combines all these actions together:

var queryOptions = new QueryOptions(CommonFileQuery.OrderByDate, new[] { ".jpg" });
var query = KnownFolders.PicturesLibrary.CreateFileQueryWithOptions(queryOptions);

var fileInfoFactory = new FileInformationFactory(query, ThumbnailMode.PicturesView);
IReadOnlyList fileInfoList = await fileInfoFactory.GetFilesAsync();
            
foreach (FileInformation fileInfo in fileInfoList)
{
   Debug.WriteLine(fileInfo.FileName);

   ImageProperties properties = fileInfo.ImageProperties;
   properties.Keywords.Add("new");
   await properties.SavePropertiesAsync();
}

FileInformation class implements IStorageItem and IStorageFile interfaces and therefore can be used to read and write, copy or delete the file. Additionally it implements IStorageItemInformation to synchronously provide file’s thumbnail and properties.

Nothing comes for free – FileInformationFactory queries are about 3 times slower that the simple indexed queries executed with the same options. The additional time is required to extract the properties and create the thumbnails – so if you don’t need all these information, the simple query will be a better option.