Side-by-side they fall

The project I'm on at work finally went to actual Windows Azure over the weekend.  This was the media server component which converts and thumbnails media that gets sent to it using a combination of ffmpeg, ImageMagick and Lokad.Cloud for message queuing.  This was our first test in the full azure environment and I found a very annoying problem.

We got ImageMagick working ages ago by putting the executables (convert.exe, identify.exe) in cloud storage, then when the application needed to use them, it would download them to local scratch storage (I'll do a bigger post on this later).  This works really well on my local machine but upon testing on actual Windows Azure convert.exe and identify.exe stopped working, quoting:

The application has failed to start because its side-by-side configuration is incorrect..  Please see the application event log ...

My first thought was to follow the application event log, so I grabbed Azure Diagnostics Manager [http://www.cerebrata.com/products/AzureDiagnosticsManager/Default.aspx] and added some code to my WorkerRole.cs/WebRole.cs OnStart...

DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
dmc.WindowsEventLog.DataSources.Add(Constants.ApplicationName);
dmc.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
DiagnosticMonitor.Start("DataConnectionString", dmc);

...and found nothing!  I couldn't get the event log and after a little bit of playing around I decided to give up path!  The problem is that every time you do small changes to azure the deploy process takes AGES - like 5 minutes or so.  So trying little things and failing them then having to wait 5 minutes between can get very frustrating.  I'm sure if I read the full how-to post (http://blog.toddysm.com/2010/05/collecting-event-logs-in-windows-azure.html) I would have worked it out but I had a feeling that the event log wouldn't tell me much regardless.

So back on track I looked into the side-by-side error and found some information about it.  Basically side-by-side errors mean that some config/assemblies are missing (http://msdn.microsoft.com/en-us/library/ms235342.aspx).  So I started along the long path of finding the missing references to ImageMagick.  By the way, the actual problem is that the VS2008 C++ Redistrib packages didn't exist on Azure whereas they existed on my system (ImageMagick download page states this at the very bottom http://www.imagemagick.org/www/binary-releases.html).  I don't think I can just install them on azure so I went about the problem by gathering all the required assemblies.

So, to investigate side-by-side issues you have to use the "sxstrace" tool.  I set up a new blank Windows 7 VM to ensure vs2008 redistributable packages weren't there, then ..

1. Run cmd elevated (Start -> type "cmd" -> right click on cmd -> Run as Adminsitrator).
2.  cd into your executable directory
3. Run "sxstrace trace -logfile:sxstrace.ctl" (without quotes)
4. In another cmd, run your side-by-side failing program (identify.exe in my case)
5. Press enter to stop tracing for sxstrace
6. The trace is a binary file that needs to be parsed.  Parse it:  sxstrace parse -logfile:sxstrace.ctl -outfile:sxstrace.txt
7. Open up sxstrace.txt and you'll find your problem.

In my case, identify.exe required a couple of dlls and some .manifest files.  You end up having to copy required manifests from c:\windows\winsxs\manifests to your executable folder then grabbing all those dlls.  Run sxstrace again (as above) to find more problems.  Here's what I ended up  with to get identify.exe and convert.exe working (my wordpress images directory isn't working so the filenames are just given below):

convert.exe
identify.exe
identify.exe.manifest (not sure if this is needed)
Microsoft.VC90.OpenMP.MANIFEST (this was renamed from the respective manifest file in c:\windows\winsxs\manifests as I was looking specifically for this name from the sxstrace log)
mscvm90.dll
msvcp90.dll
msvcr90.dll
vcomp90.dll
x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_bcb86ed6ac711f91.manifest

Happy to provide more details if anyone needs.

// Redirect Event Logs to your storage account

DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();

dmc.WindowsEventLog.DataSources.Add(Constants.ApplicationName);

dmc.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);

DiagnosticMonitor.Start("DataConnectionString", dmc)            // Redirect Event Logs to your storage account

DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();

dmc.WindowsEventLog.DataSources.Add(Constants.ApplicationName);

dmc.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);

DiagnosticMonitor.Start("DataConnectionString", dmc);

ffmpeg output

I have to do some video and audio conversions for a project I'm working on at work.  We're using ffmpeg in the project (.net) and I thought I'd write a couple of things down for my convenience:

1. ffmpeg outputs to StandardError.  This is ok, you just have to make sure you redirect the stream and read from StandardError (of course!) like so:

using (Process p = new Process()) {
p.StartInfo.FileName = "ffmpeg.exe";
p.StartInfo.UseShellExecute = false;
p.StartInfo.CreateNoWindow = true;
p.StartInfo.RedirectStandardError = true;
p.Start();
p.WaitForExit();
string response = p.StandardError.ReadToEnd();
}

2. The response given by ffmpeg are horrible!  I can't seem to find any way of formatting the response so it's pretty painful to parse.  Here's a regex that a guy at work came up with to get the length of a given video:

// To get the length, you just need to pass in the input file like so: ffmpeg -i somefile.avi
// This will give you a huge response to stderr, but the line we're interested in goes a little something like..
// Duration: 00:02:12.00, start: 0.000000, bitrate: 118 kb/s
Match match = Regex.Match(error, @"duration:.*?(?'hours'\d+):(?'minutes'\d+):(?'seconds'\d+)\.(\d+)", RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture);
if (match.Success) {
int hours = Convert.ToInt32(match.Groups["hours"].Value);
int minutes = Convert.ToInt32(match.Groups["minutes"].Value);
int seconds = Convert.ToInt32(match.Groups["seconds"].Value);
}

I barely work with regex so I have to look up what everything does each time I use it.  In the above example you'll see the ?'hours' which explicitly names the group which makes things a lot more readable.  Thanks for the example http://weblogs.asp.net/dneimke/archive/2003/05/07/6575.aspx.

azure routing

I spent this morning trying to get routing working in an azure asp.net web role (these are basically just asp.net web applications).  I thought I'd be able to use some combination of HttpHandlers and <location> to get the routing happening but alas it wasn't to be.

I remembered that I did some research into asp.net mvc apps a bit ago and remembered that they used routing so I thought I'd be able to rip it directly off that (or use the mvc libraries).  After a little digging around, routing is available to asp.net 3.5 sp1 (which is azure's current platform) just by using the standard libraries.  Actually Michael Kennedy has already posted about azure & routing perfectly - http://www.michaelckennedy.net/blog/2009/05/27/ASPNETRoutingInWindowsAzureUsingWebForms.aspx.

The crux:

• Reference System.Web.Routing and System.Web.Abstractions


• Create a Global.asax in your web role


• Modify your Global.asax like so

protected void Application_Start(object sender, EventArgs e)

{

RegisterRoutes(RouteTable.Routes);

}

public static void RegisterRoutes(RouteCollection routes)

{

var routeHandler = new WebFormRouteHandler<Page>("~/Test.aspx");

routes.Add(new Route("evidence/{environment}", routeHandler));

}

protected void Application_Start(object sender, EventArgs e)
{
RegisterRoutes(RouteTable.Routes);
}

public static void RegisterRoutes(RouteCollection routes)
{
var routeHandler = new WebFormRouteHandler<Page>("~/Test.aspx");routes.Add(new Route("evidence/{environment}", routeHandler));
}

• Create the WebFormRouteHandler

using System.Web;

using System.Web.Compilation;

using System.Web.Routing;

namespace SuperTravio

{

public class WebFormRouteHandler<T> : IRouteHandler where T : IHttpHandler, new()

{

public string VirtualPath { get; set; }

public WebFormRouteHandler(string virtualPath)

{

this.VirtualPath = virtualPath;

}

#region IRouteHandler Members

public IHttpHandler GetHttpHandler(RequestContext requestContext)

{

foreach (var aux in requestContext.RouteData.Values)

{

HttpContext.Current.Items[aux.Key] = aux.Value;

}

return (VirtualPath != null)

? (IHttpHandler)BuildManager.CreateInstanceFromVirtualPath(VirtualPath, typeof(T))

: new T();

}

#endregion

}

}

• Create a Url routing handler that overrides IIS7's default action

using System.Web;

using System.Web.Routing;

namespace Supertravio

{

public class RoutingHandler : UrlRoutingHandler

{

protected override void VerifyAndProcessRequest(IHttpHandler httpHandler, HttpContextBase httpContext)

{

}

}

}

• Update your web.config to use the handler (from Michael's post)

<system.webserver>

<modules runallmanagedmodulesforallrequests="true">

...

<add name="UrlRoutingModule" type="System.Web.Routing.UrlRoutingModule,

System.Web.Routing, Version=3.5.0.0,

Culture=neutral,

PublicKeyToken=31BF3856AD364E35">

</add>

<handlers>

<add name="UrlRoutingHandler" precondition="integratedMode" verb="*" path="UrlRouting.axd" type="Supertravio.RoutingHandler, Supertravio">

</add>

</handlers>

</modules></system.webserver>

You're done!

The only thing now is accessing the routed parts.  So in my example (new Route("evidence/{environment}") I would get the parts from my page using (string)HttpContext.Current.Items["environment"].

Setting up azure

I had to set up azure a couple of times and every time I forget how to get it all up and running, so here's the business:

http://blogs.msdn.com/jnak/archive/2009/04/20/installing-the-windows-azure-tools-and-sdk-the-easy-way.aspx