Recently, I have run into an interesting issue with one of my websites – it runs in Azure App Service and I was using automated deployments from VSTS Azure DevOps.
After the website was deployed, it didn’t start – I was getting HTTP 502.
Diagnostics
When I deploy something into Azure App Service and the app doesn’t start, I go to the Kudu console first (https://nameofyoursite.scm.azurewebsites.net) and look in LogFiles/eventlog.xml.
If there is a problem with app startup (configuration error, missing DLLs or an exception thrown during the initialization of the application), there is a chance the error will be in this file.
If you are using Application Insights and the exception occurs on startup, it will probably not be recorded because the Application Insights DLLs may not be loaded and initialized.
You can also turn on filesystem logging in Azure portal to find more details.
Cannot create directory? What?
A quick look in eventlog.xml using Kudu console told me that I am getting FileNotFoundException (Could not find file 'D:\home\site\wwwroot\Temp'.) from Directory.CreateDirectory.
It was quite strange – the path was correct and the function should actually create that directory instead of complaining that the path doesn’t exist.
In general, it is not the best idea for a web app to write in the filesystem, but most web apps does this, at least they write some log files, store uplaoded files temporarily before they are processed and so on.
After a few minutes, I discovered another weird thing – when I was browsing the wwwroot folder in Kudu, the Temp folder was not there, and when I tried to create it using Kudu, I got the following error:
409 Conflict: Cannot delete directory. It is either not empty or access is not allowed.
What? I was creating a directory, not deleting anything.
I tried to use mkdir Temp in the command line, but got the following:
The system cannot find the file specified.
Desperate enough, I tried to connect using FTP and the folder was there! I tried to delete it, but I got the same results from the app and from the Kudu.
Then I noticed that FTP shows me old versions of some files. So the app must have been running from a different folder. I double checked the FTP and Kudu addresses, but they were the same.
What was even more strange – the previous version of the web app did the same writes in the filesystem and it worked. The startup code didn’t change at all and the app worked normally before the deployment.
What has changed? What have I done?
Azure App Service Deploy Task
The only difference was the deployment process. The previous version of the website was deployed few months ago directly from Visual Studio.
This time, I tried to deploy using Azure DevOps which has a very nice deployment task for that. It worked for the test site so I just create a different environment for production and deployed there.
I have looked at the definition of the deployment task, but haven’t found anything unusual – it was quite straight-forward – take the build artifacts and push them in the Azure App Service.
What now? Because it was a production site with some traffic, I decided to just deploy from Visual Studio to fix the error quickly, and then dive into the cause of the issue. So I hit Publish and couldn’t believe the error message:
Invalid access to memory location.
I started googling and finally found the answer.
Run from Package
The 4.* version of the Azure App Service Deploy task is using Run from Package application mode by default, which means that it uploads a ZIP file with the app (they call it Zip Deploy) and sets WEBSITE_RUN_FROM_ZIP application setting to 1.
The application then runs from the ZIP package - there is a virtual file system which makes the application and Kudu console see the contents of the ZIP package in the wwwroot folder.
The virtual file system it is not used when you connect using FTP, so that’s why I was seing different files in the folder.
And because the application runs from the ZIP package, it cannot write to its filesystem. Sadly, the error messages produced by I/O functions are not helpful.
Since most web apps I have seen write in their filesystem, this is quite significant change of behavior, and making it a default option in Azure DevOps deployment task can lead to a lot of confusion.
I didn’t know about this feature at all, and what is more, the setting is hidden in VSTS task so I didn’t notice it. You need to expand the Additional Deployment Options section and click on the Select deployment method checkbox, which is unchecked by default. Only after these two clicks, you can see the dropdown with deployment methods – ZipDeploy is the default one.
I needed to change it to use WebDeploy so the application files will be stored as normal files and the application can write in the filesystem like it could before.
And don’t forget to remove the WEBSITE_RUN_FROM_ZIP application setting, otherwise the deployment will fail with Invalid access to memory location error.