Azure big upload take 2

Since posting the last big upload post we realised there was actually a 100MB upload limit in Azure.  This isn't 100MB per user.. this is 100MB per server.  So if 2 users upload 55MB each say, they will most likely both fail.  I think this is because the location for the temporary files only has 100MB available.  Sure, you can configure another location in web.config but that's not really possible when you can only get a writable location at runtime.  We emailed Azure support and it's been about a week with only this response so far:

"...So far the problem you have reported is a known bug and there is no fix for the issue besides the following workarounds:

1. Use 3rd party ASP component which does not use asp.net temp folder


2. Use silverlight


3. Create your own asp.net upload component which does not use asp.net temp folder..."

So.. where to now!?  I did a bit of research and came across a great post that allows you to bypass asp.net upload handling -  http://dimebrain.com/2010/01/large-or-asynchronous-file-uploads-in-asp-net-mvc.html.  The only problem with this solution is that it doesn't appear to handle the form fields which we also need.  I actually tried it too and found it threw an exception pretty early in the piece and because it didn't quite cover what we needed I made a version which handles any number of files, a file size limit (although this only applies to the size of the stream itself) and any number of form fields.

I also tried to make this in an ActionFilter, eg ...

[BigUpload(100)]
public ActionResult Upload()

... but it appeared that using an ActionFilter was actually forcing asp.net to start handling the stream itself which is what we didn't want.  I'm not 100% sure whether it was the ActionFilter that caused this or whether it was some call inside the ActionFilter that caused it to deal with the stream itself.

So the solution.  First the usage:

public ActionResult Upload() {
// I have a 20GB local resource configured for this web role called "UploadStorage"
var temporaryStorageLocation = RoleEnvironment.GetLocalResource("UploadStorage").RootPath;
// work with the underlying connection..
var context = ControllerContext.HttpContext;
var provider = (IServiceProvider)context;
var workerRequest = (HttpWorkerRequest)provider.GetService(typeof(HttpWorkerRequest));

var processor = new BigUploadProcessor(workerRequest, temporaryStorageLocation);
processor.UploadLength = 100 * 1024 * 1024; //100MB
processor.HandleRequest(context, context.Request.ContentEncoding);

// at this point you can now use processor.Fields["fieldName"] or processor.Files["fieldName"]

Then you just need the guts: BigUpload .  I've tested it in firefox, chrome and internet explorer and haven't had any issues but please comment if you try it and run into any problems.

This is provided for information only, use at your own risk.

Auto Draft

Auto Draft

Azure uploading big files into blob storage

Update: Please see my updated azure big files post.

Azure is all about big files, solving big computational problems and being able to expand quickly without needing your own infrastructure.

Why oh why then is uploading largish files (anywhere from 4MB in size) to blob storage such a pain?  This is my third go and I'm still not convinced it's going to work.  Here's what I've done so far to enable largish files to be uploaded:

1. Added a 20 minute execution timeout and 100MB file size maximum in web.config - <httpRuntime executionTimeout="1200" maxRequestLength="2000000" />


2. Then on the CloudBlobClient I set the Timeout = new TimeSpan(2, 0, 0); (2 hours) and the ParallelOperationThreadCount = 2.


3. 
   
   It looks like you also have to add another web.config change as below:
   
   
   <system.webServer>
   
   
   <!-- This is a setting from this forum http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/f4575746-a695-40ff-9e49-ffe4c99b28c7/ -->
   
   
   <security>
   
   
   <requestFiltering>
   
   
   <!-- 100MB -->
   
   
   <requestLimits maxAllowedContentLength="2000000000"></requestLimits>
   
   ...

I've seen examples where people manually chunk the upload into a number of streams of 4MB size which I'm really hoping I don't have to do.  Fingers crossed.

Thanks wot u do - http://wotudo.net/blogs/wotudo/archive/2010/02/16/copying-files-to-windows-azure-blob-storage.aspx.

Update: Added point 3.  In the process of seeing if it works now.

Update: man this whole thing seems a little flaky.  The above seems to work now but let's see..

azure nhibernate

I'm currently setting up our project with nhibernate and have been trying to work out a couple of things.  Firstly, the application is spread over a number of assemblies and the configuration loading ended up in our Project.Framework assembly.

If I tried embedding our hibernate.cfg.xml in the Project.Web (Web Role) assembly, the configuration loading wouldn't work.  I ended up putting the hibernate.cfg.xml in App_Data in the Project.Web assembly, then when I load my config:

var configuration = new Configuration();

var configFilename = "hibernate.cfg.xml";

if (HttpContext.Current != null)

{

configFilename= HttpContext.Current.Server.MapPath("~/App_Data/" + configFilename);

}

configuration.Configure(configFilename);

configuration.AddAssembly("Project.Data");

return configuration;

In actual fact I'm using hibernate.[dev|tst|prd].cfg.xml and I'm wondering if this is the reason it wouldn't load.  In any case, this way seems to be secure so that's all I wanted.

I'm using a modified ISessionBuilder from Jeffrey's blog here http://jeffreypalermo.com/blog/use-this-nhibernate-wrapper-to-keep-your-repository-classes-simple/.

web.config tips

I'm relatively new to the whole web.config in asp.net, being more used to working with ye olde' WEB-INF/web.xml or perhaps even the portlet.xml. 

To do page timeouts in asp.net you can use the <httpRuntime executionTimeout="<timeout in seconds>"/>.

This is a great article about all things web.config http://articles.sitepoint.com/article/web-config-file-demystified

Azure error handling

This isn't so much about error handling in azure as it is error handling in asp.net.  Ok the title kinda lied.. :) 

Adding error handling in asp.net is easy:

1. Create an error page in your application (eg error/Error.aspx). 


2. In the <customErrors/> config, set the defaultRedirect="error/Error.aspx" and voila!


3. You have a couple of opportunities to handle the error (when it happens, in the page, or in the application):
   
   
   
   • In the page: public void Page_Error(object sender, EventArgs e)
   
   
   • In the application (Global.asax: protected void Application_Error(object sender, EventArgs e)
   
   
   
   


4. Then you can access the actual exception via Server.GetLastError().GetBaseException();

So at work our exceptions get mailed.  Since azure currently doesn't have mail servers, we opted to use SendGrid which seemed quite ok.

Full guide here http://support.microsoft.com/kb/306355.