We have created an Azure WebJob to do some scheduled database cleanup for our WebAPI project. Now we want to display the latest job runs in our own management web app to monitor how the cleanup is going every day. How can I get the latest 50 function calls including their corresponding output logs and durations for a given WebJob?
Some background for this requirement:
The cleanup process we came up with is very simple but we suspect it will start to bottleneck us in the near future, so we wanted to monitor how long it takes to run each day to proactively redesign it using a more scalable approach when needed. Ideally I'd like to get the last 50 or so runs and generate a graph showing the time it took to execute over that period of time.
The first thing we thought about was to just create our own database and wrap each execution with a Stopwatch then save the duration to our database. We could then just query this database and build the graph that way.
But since this was the very first time we used WebJobs, we quickly found out that most of the stuff we wanted to log is already logged automatically by the WebJobs SDK. Things like start times, duration, function name, etc, all exist already in the logs and would be enough for us to build our own display around.
The problem then is how to query those logs from our MVC project in a suitable format. I could of course read from the blob storage directly and join the data manually, but I was looking for a more high level API. To me the storage containers used by the WebJobs are an implementation detail that could change in the future.
When searching for that, I found numerous ways to get to the data, but I feel none of them are really what I was looking for.
There is the
Get-AzureWebsiteLogPowerShell cmdlet that returns the data for a set of runs for a job. This looks ok, but I wanted something callable directly from .Net, and I'd like to see the console outputs for the action too, and I can't seem to find those with this cmdlet.There is a WebAPI to query the logs for a given job. That one seems to be returning the exact same response as the PowerShell version (the PowerShell cmdlet is probably calling the same REST endpoint under the covers). So the same problems apply.
By looking at the actual files created on the blob storage account, I can get access to the output logs for each run by inspecting the /azure-webjobs-hosts/output-logs folder and correlating the id there with the values on the /azure-webjobs-dashboard/functions/instances/ folder, so I suspect it would be possible somehow to get them using a more high level API. Also, if such API was available, I wouldn't need to worry about having to create type safe models for each API response and things like that.
