Azure uploading big files into blob storage

Update: Please see my updated azure big files post.

Azure is all about big files, solving big computational problems and being able to expand quickly without needing your own infrastructure.

Why oh why then is uploading largish files (anywhere from 4MB in size) to blob storage such a pain?  This is my third go and I'm still not convinced it's going to work.  Here's what I've done so far to enable largish files to be uploaded:

1. Added a 20 minute execution timeout and 100MB file size maximum in web.config - <httpRuntime executionTimeout="1200" maxRequestLength="2000000" />


2. Then on the CloudBlobClient I set the Timeout = new TimeSpan(2, 0, 0); (2 hours) and the ParallelOperationThreadCount = 2.


3. 
   
   It looks like you also have to add another web.config change as below:
   
   
   <system.webServer>
   
   
   <!-- This is a setting from this forum http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/f4575746-a695-40ff-9e49-ffe4c99b28c7/ -->
   
   
   <security>
   
   
   <requestFiltering>
   
   
   <!-- 100MB -->
   
   
   <requestLimits maxAllowedContentLength="2000000000"></requestLimits>
   
   ...

I've seen examples where people manually chunk the upload into a number of streams of 4MB size which I'm really hoping I don't have to do.  Fingers crossed.

Thanks wot u do - http://wotudo.net/blogs/wotudo/archive/2010/02/16/copying-files-to-windows-azure-blob-storage.aspx.

Update: Added point 3.  In the process of seeing if it works now.

Update: man this whole thing seems a little flaky.  The above seems to work now but let's see..

azure nhibernate

I'm currently setting up our project with nhibernate and have been trying to work out a couple of things.  Firstly, the application is spread over a number of assemblies and the configuration loading ended up in our Project.Framework assembly.

If I tried embedding our hibernate.cfg.xml in the Project.Web (Web Role) assembly, the configuration loading wouldn't work.  I ended up putting the hibernate.cfg.xml in App_Data in the Project.Web assembly, then when I load my config:

var configuration = new Configuration();

var configFilename = "hibernate.cfg.xml";

if (HttpContext.Current != null)

{

configFilename= HttpContext.Current.Server.MapPath("~/App_Data/" + configFilename);

}

configuration.Configure(configFilename);

configuration.AddAssembly("Project.Data");

return configuration;

In actual fact I'm using hibernate.[dev|tst|prd].cfg.xml and I'm wondering if this is the reason it wouldn't load.  In any case, this way seems to be secure so that's all I wanted.

I'm using a modified ISessionBuilder from Jeffrey's blog here http://jeffreypalermo.com/blog/use-this-nhibernate-wrapper-to-keep-your-repository-classes-simple/.

web.config tips

I'm relatively new to the whole web.config in asp.net, being more used to working with ye olde' WEB-INF/web.xml or perhaps even the portlet.xml. 

To do page timeouts in asp.net you can use the <httpRuntime executionTimeout="<timeout in seconds>"/>.

This is a great article about all things web.config http://articles.sitepoint.com/article/web-config-file-demystified

Azure error handling

This isn't so much about error handling in azure as it is error handling in asp.net.  Ok the title kinda lied.. :) 

Adding error handling in asp.net is easy:

1. Create an error page in your application (eg error/Error.aspx). 


2. In the <customErrors/> config, set the defaultRedirect="error/Error.aspx" and voila!


3. You have a couple of opportunities to handle the error (when it happens, in the page, or in the application):
   
   
   
   • In the page: public void Page_Error(object sender, EventArgs e)
   
   
   • In the application (Global.asax: protected void Application_Error(object sender, EventArgs e)
   
   
   
   


4. Then you can access the actual exception via Server.GetLastError().GetBaseException();

So at work our exceptions get mailed.  Since azure currently doesn't have mail servers, we opted to use SendGrid which seemed quite ok.

Full guide here http://support.microsoft.com/kb/306355.

CloudBlockBlob.DownloadToFile()

I've been using CloudBlockBlob.DownloadToFile() without a hitch for the last couple of months but I started running parts of the project locally to do some debugging and it looks like DownloadToFile has a bit of a problem with it.  When I try DownloadToFile on a CloudBlockBlob that isn't small small (it's actually 64MB), I get a System.IO.IOException: "Unable to read data from the transport connection: The connection was closed."

Looks like you need to use the stream version:

// This will override anything that already exists,

using (FileStream fs = new FileStream(filename, FileMode.Create))
{

// I'm using a one-hour timeout

fileBlob.DownloadToStream(fs, new BlobRequestOptions() { Timeout = new TimeSpan(1, 0, 0)});

}

This guy also had the same problem http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/649ee4a0-8d38-4cd7-aba5-1c34cd033865

Update: I did a bit of testing with the Timeout BlobRequestOptions and the TimeSpan you set is more of a guideline.. ;)  I set the timeout to be 1 second and it took about 10 seconds to actually timeout.  I'm guessing that they stream a certain amount (I think it was 4096kb) then check if the timeout has passed, then read a bit more if it hasn't.  In any case, it's working now.

Update 2: Stupidly I wasn't closing the file stream which caused the file to be cut short!  argh!  Anyway, wrap it in a "using" like usual to call the Close().