Replacement Task Scheduler for Sitecore

In this article we discuss some of the challenges with the out-of-the-box Task Scheduler included with Sitecore and see how you can replace it with Hangfire, a product to perform background processing for .Net applications.

Update: A NuGet package is available for download here.

There are ton of articles describing the Task Scheduler and oftentimes cover the same information. Below are few to get you started:

• Sitecore Scheduled Task – Schedule time format and other quirks
• Scheduled Tasks in Sitecore: Everything's Relative
• Sitecore PowerShell Extensions Integration with Tasks

Oddly the only things I can find on the Sitecore docs site is from the old SDN. I'll skip linking that here because it is likely to break.

Some of the issues you'll find with the Task Scheduler is the inability to run at a specific time and if Sitecore shuts down the missed tasks are likely to run immediately following startup. With the use of Hangfire we'll address both issues. The format of the schedule field is also a bit crazy and so we'll add to the complexity by including support for the cron format.

So why not SiteCron? You should use it. If however you can't use it, don't want to use it, or simply can't make up your mind then feel free to give this a try.

Here is a quick breakdown of what we'll build:

1. Pipeline processor inheriting from Sitecore.Owin.Pipelines.Initialize.InitializeProcessor which should give us access to IAppBuilder. Here we'll register Hangfire and handle scheduling of jobs.
2. Configuration patch to disable the agent used for scheduled tasks and to register our new processor. There are multiple agents used for scheduled tasks, so at the moment we'll only focus on the one that runs for master in the Standalone and ContentManagement roles.

The important part

Sitecore configuration patch to register the code:

	<?xml version="1.0" encoding="utf-8" ?>
	<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/" xmlns:role="http://www.sitecore.net/xmlconfig/role/">
	<sitecore role:require="Standalone or ContentManagement">
	<pipelines>
	<owin.initialize>
	<processor type="Scms.Foundation.Scheduling.Pipelines.PrecisionScheduler, Scms.Foundation">
	<StartupDelaySeconds>120</StartupDelaySeconds>
	<RefreshSchedule>*/2 * * * *</RefreshSchedule>
	</processor>
	</owin.initialize>
	</pipelines>
	<scheduling>
	<!-- Replaced by the PrecisionScheduler -->
	<agent name="Master_Database_Agent">
	<patch:attribute name="interval" value="00:00:00" />
	</agent>
	</scheduling>
	</sitecore>
	</configuration>

view raw Scms.Foundation.Scheduling.config hosted with ❤ by GitHub

Explanation:

• Register processor in the own.initialize pipeline. The first option allows you to provide the amount of time to delay running scheduled tasks after Sitecore starts up; this is extremely helpful if you want to disable jobs and need some extra time or the jobs are process-intensive and you want to give Sitecore time to warm up. The second option allows you to adjust the frequency in which schedules configured in Sitecore (from the Content Editor) are updated in Hangfire; this is important for cases where Admins create/update/remove scheduled tasks.
• Disable the the Master_Database_Agent so the old scheduler doesn't run.

Sitecore processor:

	using Hangfire;
	using Hangfire.MemoryStorage;
	using Hangfire.Storage;
	using Microsoft.Extensions.DependencyInjection;
	using Sitecore;
	using Sitecore.Abstractions;
	using Sitecore.Data;
	using Sitecore.Data.Items;
	using Sitecore.DependencyInjection;
	using Sitecore.Diagnostics;
	using Sitecore.Jobs;
	using Sitecore.Owin.Pipelines.Initialize;
	using Sitecore.Tasks;
	using System;
	using System.Collections.Generic;
	using System.Linq;
	using System.Text.RegularExpressions;
	namespace Scms.Foundation.Scheduling.Pipelines
	{
	public class PrecisionScheduler : InitializeProcessor
	{
	private const string SCHEDULE_DATABASE = "master";
	public string RefreshSchedule { get; set; } = "*/2 * * * *";
	public int StartupDelaySeconds { get; set; } = 15;
	private static readonly RecurringJobOptions JobOptions = new RecurringJobOptions() { TimeZone = TimeZoneInfo.Local, MisfireHandling = MisfireHandlingMode.Ignorable };
	private static void LogMessage(string message)
	{
	Log.Info($"[PrecisionScheduler] {message}", nameof(PrecisionScheduler));
	}
	public override void Process(InitializeArgs args)
	{
	var app = args.App;
	app.UseHangfireAspNet(() =>
	{
	GlobalConfiguration.Configuration.UseMemoryStorage();
	return new[] { new BackgroundJobServer() };
	});
	LogMessage("Starting up precision scheduler.");
	BackgroundJob.Schedule(() => Initialize(RefreshSchedule), TimeSpan.FromSeconds(StartupDelaySeconds));
	}
	private static string GenerateMultiDayCronExpression(TimeSpan runTime, List<DayOfWeek> daysToRun)
	{
	var castedDaysToRun = daysToRun.Cast<int>().ToList();
	return $"{ParseCronTimeSpan(runTime)} * * {ParseMultiDaysList(castedDaysToRun)}";
	}
	private static string ParseCronTimeSpan(TimeSpan timeSpan)
	{
	if (timeSpan.Days > 0)
	{
	//At HH:mm every day.
	return $"{timeSpan.Minutes} {timeSpan.Hours}";
	}
	else if (timeSpan.Hours > 0)
	{
	//At m minutes past the hour, every h hours.
	return $"{timeSpan.Minutes} */{timeSpan.Hours}";
	}
	else if (timeSpan.Minutes > 0)
	{
	//Every m minutes.
	return $"*/{timeSpan.Minutes} *";
	}
	return $"*/30 *";
	}
	private static string ParseMultiDaysList(List<int> daysToRun)
	{
	if (daysToRun.Any() && daysToRun.Count == 7) return "*";
	return string.Join(",", daysToRun);
	}
	private static List<DayOfWeek> ParseDays(int days)
	{
	var daysOfWeek = new List<DayOfWeek>();
	if (days <= 0) return daysOfWeek;
	if (MainUtil.IsBitSet((int)DaysOfWeek.Sunday, days))
	{
	daysOfWeek.Add(DayOfWeek.Sunday);
	}
	if (MainUtil.IsBitSet((int)DaysOfWeek.Monday, days))
	{
	daysOfWeek.Add(DayOfWeek.Monday);
	}
	if (MainUtil.IsBitSet((int)DaysOfWeek.Tuesday, days))
	{
	daysOfWeek.Add(DayOfWeek.Tuesday);
	}
	if (MainUtil.IsBitSet((int)DaysOfWeek.Wednesday, days))
	{
	daysOfWeek.Add(DayOfWeek.Wednesday);
	}
	if (MainUtil.IsBitSet((int)DaysOfWeek.Thursday, days))
	{
	daysOfWeek.Add(DayOfWeek.Thursday);
	}
	if (MainUtil.IsBitSet((int)DaysOfWeek.Friday, days))
	{
	daysOfWeek.Add(DayOfWeek.Friday);
	}
	if (MainUtil.IsBitSet((int)DaysOfWeek.Saturday, days))
	{
	daysOfWeek.Add(DayOfWeek.Saturday);
	}
	return daysOfWeek;
	}
	public static void RunSchedule(ID itemId)
	{
	var database = ServiceLocator.ServiceProvider.GetRequiredService<BaseFactory>().GetDatabase("master", true);
	var item = database.GetItem(itemId);
	if (item == null)
	{
	LogMessage($"Removing background job for {itemId}.");
	RecurringJob.RemoveIfExists(itemId.ToString());
	return;
	}
	var jobName = $"{nameof(PrecisionScheduler)}-{itemId}";
	var runningJob = JobManager.GetJob(jobName);
	if (runningJob != null && runningJob.Status.State == JobState.Running)
	{
	LogMessage($"Background job for {itemId} is already running.");
	return;
	}
	LogMessage($"Running background job for {itemId}.");
	var scheduleItem = new ScheduleItem(item);
	var jobOptions = new DefaultJobOptions(jobName, "scheduling", "scheduler", Activator.CreateInstance(typeof(JobRunner)), "Run", new object[] { ID.Parse(itemId) });
	JobManager.Start(jobOptions);
	}
	public static void Initialize(string refreshSchedule)
	{
	ManageJobs(true);
	RecurringJob.AddOrUpdate(nameof(ManageJobs), () => ManageJobs(false), refreshSchedule, JobOptions);
	}
	public static void ManageJobs(bool isStartup)
	{
	var database = ServiceLocator.ServiceProvider.GetRequiredService<BaseFactory>().GetDatabase(SCHEDULE_DATABASE, true);
	var descendants = database.SelectItems($"/sitecore/system/tasks/schedules//*[@@templateid='{TemplateIDs.Schedule}']");
	var schedules = new Dictionary<string, string>();
	foreach (var item in descendants)
	{
	if (item.TemplateID != TemplateIDs.Schedule) continue;
	var itemId = item.ID.ToString();
	var schedule = GetSchedule(item);
	if (string.IsNullOrEmpty(schedule)) continue;
	schedules.Add(itemId, schedule);
	}
	var jobs = JobStorage.Current.GetConnection().GetRecurringJobs();
	var existingJobs = new List<string>();
	foreach (var job in jobs)
	{
	if (!ID.IsID(job.Id)) continue;
	var itemId = job.Id;
	if (!schedules.ContainsKey(itemId))
	{
	LogMessage($"Removing {itemId} from recurring schedule.");
	RecurringJob.RemoveIfExists(itemId);
	continue;
	}
	var item = database.GetItem(itemId);
	var schedule = GetSchedule(item);
	if (string.IsNullOrEmpty(schedule))
	{
	LogMessage($"Removing {itemId} from recurring schedule with invalid expression.");
	RecurringJob.RemoveIfExists(itemId);
	continue;
	}
	if (!string.Equals(job.Cron, schedule, StringComparison.InvariantCultureIgnoreCase))
	{
	LogMessage($"Updating {itemId} with a new schedule '{schedule}'.");
	RecurringJob.AddOrUpdate($"{itemId}", () => RunSchedule(ID.Parse(itemId)), schedule, JobOptions);
	}
	existingJobs.Add(itemId);
	}
	var missingJobs = schedules.Keys.Except(existingJobs);
	foreach (var missingJob in missingJobs)
	{
	var itemId = missingJob;
	var item = database.GetItem(itemId);
	var schedule = GetSchedule(item);
	LogMessage($"Registering recurring job for {itemId} with schedule '{schedule}'.");
	RecurringJob.AddOrUpdate($"{itemId}", () => RunSchedule(ID.Parse(itemId)), schedule, JobOptions);
	}
	if (isStartup)
	{
	var recurringJobs = JobStorage.Current.GetConnection().GetRecurringJobs();
	if (recurringJobs == null) return;
	foreach (var recurringJob in recurringJobs)
	{
	if (!ID.IsID(recurringJob.Id)) continue;
	var itemId = recurringJob.Id;
	if (!schedules.ContainsKey(itemId)) continue;
	var item = database.GetItem(itemId);
	var scheduleItem = new ScheduleItem(item);
	var missedLastRun = (recurringJob.NextExecution - scheduleItem.LastRun) > TimeSpan.FromHours(24);
	if (missedLastRun)
	{
	LogMessage($"Running missed job {itemId}.");
	var jobName = $"{nameof(PrecisionScheduler)}-{itemId}";
	var jobOptions = new DefaultJobOptions(jobName, "scheduling", "scheduler", Activator.CreateInstance(typeof(JobRunner)), "Run", new object[] { ID.Parse(itemId) });
	JobManager.Start(jobOptions);
	}
	}
	}
	}
	private static string GetSchedule(Item item)
	{
	var schedule = item.Fields[ScheduleFieldIDs.Schedule].Value;
	if (string.IsNullOrEmpty(schedule)) return string.Empty;
	if (Regex.IsMatch(schedule, @"^(((\d+,)+\d+|(\d+|\*(\/|-)\d+)|\d+|\*)\s?){5,7}$", RegexOptions.Compiled))
	{
	return schedule;
	}
	var recurrence = new Recurrence(schedule);
	if (recurrence.Days == DaysOfWeek.None ||
	recurrence.Interval == TimeSpan.Zero ||
	recurrence.InRange(DateTime.UtcNow) != true) return string.Empty;
	return GenerateMultiDayCronExpression(recurrence.Interval, ParseDays((int)recurrence.Days).ToList());
	}
	}
	public class JobRunner
	{
	public void Run(ID itemId)
	{
	var database = ServiceLocator.ServiceProvider.GetRequiredService<BaseFactory>().GetDatabase("master", true);
	var item = database.GetItem(itemId);
	if (item == null) return;
	var scheduleItem = new ScheduleItem(item);
	scheduleItem.Execute();
	}
	}
	}

view raw Scms.Foundation.Scheduling.Pipelines.PrecisionScheduler.cs hosted with ❤ by GitHub

Explanation:

• You'll need references to Hangfire.AspNet, Hangfire.Core, and Hangfire.MemoryStorage found on Nuget. I went with in-memory storage because I like things simple and having to setup connections to a database and custom tables sounds like a pain. Also, I feel like looking at the LastRun field was enough to satisfy what I needed.
• In the Process method we do the basic configuration for Hangfire. I don't care about the dashboard like described here and here. Once again, I like things simple and adding another thing to maintain is just not for me. If you want to then go knock yourself out. We also schedule a fire-and-forget background job to initialize the scheduler which includes registering Scheduled Tasks defined in Sitecore.
• Once the first background job runs, which may execute tasks that missed their scheduled run time, the recurring jobs are configured. Any scheduled task with an empty/invalid schedule is ignored. The creation of the recurring jobs are chained to the Initialize method so we can ensure that anything missed runs before we schedule others. This may be a problem if you have really long running jobs.
• The default schedule format {start timestamp}|{end timestamp}|{days to run bit pattern}|{interval} is compatible but converted to a cron schedule format when registered (see GetSchedule method). The interval may be converted to a precise time when using the daily TimeSpan format. For example, the value 1.04:30:00 will run daily at 4:30 AM. The conversion code produces a cron format 30 4 * * * which is what ultimately gets provided to Hangfire. If you want to replace the default schedule format with a cron schedule format go right ahead as both formats are supported. The only problem I saw with this is when using the SPE Task Manager because the tool doesn't know anything about cron.

A simple report can be written with Sitecore PowerShell Extensions to display the current configured recurring jobs.

	$connection = [Hangfire.JobStorage]::Current.GetConnection()
	$recurringJobs = [Hangfire.Storage.StorageConnectionExtensions]::GetRecurringJobs($connection)
	$props = @{
	Title = "Hangfire Recurring Jobs"
	InfoTitle = "Recurring Jobs Report"
	InfoDescription = "This report provides details on the currently scheduled recurring jobs."
	PageSize = 25
	Property = @(
	"Id",
	@{Label="Task"; Expression={Get-Item -Path "master:" -ID $_.ID | Select-Object -ExpandProperty Name}},
	"Cron",
	@{Label="NextExecution (Local)"; Expression={$_.NextExecution.ToLocalTime()} },
	"LastJobState",
	@{Label="LastExecution (Local)"; Expression={$_.LastExecution.ToLocalTime()} },
	"TimeZoneId",
	"Error",
	"RetryAttempt"
	)
	}
	$recurringJobs | Show-ListView @props
	Close-Window

view raw HangfireRecurringJobs.ps1 hosted with ❤ by GitHub

We've had it running for quite some time now and it has been a game changer for us. Think of all the scheduled tasks that seem to randomly run during deployments or competing with processing power needed by Content Editors.

Give it a try and let me know how it works out for you on your projects. Feedback welcome.

More Hangfire:

• Publishing Notification with Hangfire
  
• Historic Log with Hangfire
  

Disclaimer: After sharing this post I was reminded of an important distinction one should make with bolting on more features to the Sitecore platform. I found Hangfire helped solve an issue we were having with our scheduled tasks. Every day we run tasks that execute SPE scripts and these can be quite CPU intensive. The data belongs in Sitecore and the tasks need to be run outside of business hours (late night or early morning). As a separate application we built a sync with Quartznet using dotnet 6 (latest LTS version at the time) which dramatically improved the developer experience, performance, and maintainability.