Shift Server

The server component retrieves available jobs and executes commands from clients. The server is a simple .NET library, not an executable and it needs to run inside a .NET container application, such as ASP.NET, WPF, WinForm, Azure WebJob, or Windows service. Each Shift server can have multiple workers, multiple server apps can also run side by side safely with unique process ID.

The Shift server polls the job queue in regular interval and retrieves ready to run jobs based on first-in, first-out method. The FIFO method can be interrupted based on run-now command from clients, the server will attempt to retrieve run-now jobs first and without regards to the FIFO creation order. The workers in each server run in order according to their creation ID, which starts from 1. For example, with 2 workers, the ID will be given as #1 and #2. When the server attempts to retrieve jobs, the first worker #1 will try to pull as many jobs up to the MaxRunnableJobs setting, then the second worker #2 will do the same thing. The server actions are always done in order according to the worker ID, so worker #1, #2, #3, etc. and it repeats itself automatically for every actions (pulling and running jobs, stopping jobs, clean up, etc.).

Two runnable server apps projects are included as quick start templates:

• Shift.WinService is the standalone Windows service server component, multiple services can be installed in the same server.
• Shift.WebJob is the Azure WebJob component that can be easily deployed to Azure cloud environment, multiple web jobs can also be deployed to multiple App Services. If you're using Azure, it is highly recommended to locate the Azure SQL and Azure Redis within the same region as the web jobs.

To enable the Shift.Server apps to run jobs shifted from the client apps, the server apps are required to:

• Have all assembly dependencies loaded by the Shift server on start.

There are two dynamic ways to load dependencies. You can set the path location of the assemblies in AssemblyFolder configuration or list assembly locations in the assembly list text file set in AssemblyListPath. With dynamic loading, any assembly files changes only requires simple Shift server restart.
You can also reference assembly files directly before server compilation, however any changes may require recompiling your entire server app.

An example of the possible content of the assembly list text file:

C:\ShiftServer\dependencies\demo.businesslayer.dll
dependencies\demo.datalayer.dll
dependencies\demo.joblogic.*.dll
dependencies\contoso\contoso.timers.dll
..\reporting\PDFGenerator.dll

• Initialize all required storage and Shift server settings during startup.
  Use .NET config file, database, or Azure Key Vault to store the global settings. Initialize Shift server in ASP.NET global.asax or main method using those configurations.

var serverConfig = new Shift.ServerConfig();
serverConfig.DBConnectionString = ConfigurationManager.ConnectionStrings["ShiftDBConnection"].ConnectionString;
serverConfig.DBAuthKey = ConfigurationManager.AppSettings["DocumentDBAuthKey"];
serverConfig.EncryptionKey = ConfigurationManager.AppSettings["ShiftEncryptionParametersKey"]; //optional, will encrypt parameters in DB if exists
serverConfig.MaxRunnableJobs = Convert.ToInt32(ConfigurationManager.AppSettings["MaxRunnableJobs"]); 
serverConfig.ProcessID = ConfigurationManager.AppSettings["ShiftPID"];
serverConfig.Workers = Convert.ToInt32(ConfigurationManager.AppSettings["ShiftWorkers"]);

serverConfig.StorageMode = ConfigurationManager.AppSettings["StorageMode"];
var progressDBInterval = ConfigurationManager.AppSettings["ProgressDBInterval"];
if(!string.IsNullOrWhiteSpace(progressDBInterval))
	serverConfig.ProgressDBInterval = TimeSpan.Parse(progressDBInterval); //Interval when progress is updated in main DB

var autoDeletePeriod = ConfigurationManager.AppSettings["AutoDeletePeriod"];
serverConfig.AutoDeletePeriod = string.IsNullOrWhiteSpace(autoDeletePeriod) ? null : (int?)Convert.ToInt32(autoDeletePeriod);
serverConfig.AutoDeleteStatus = new List<JobStatus?> { JobStatus.Completed }; //Auto delete only the jobs that had been Completed

serverConfig.ForceStopServer = Convert.ToBoolean(ConfigurationManager.AppSettings["ForceStopServer"]); 
serverConfig.StopServerDelay = Convert.ToInt32(ConfigurationManager.AppSettings["StopServerDelay"]);

//serverConfig.ServerTimerInterval = Convert.ToInt32(ConfigurationManager.AppSettings["ServerTimerInterval"]);     
//serverConfig.ServerTimerInterval2 = Convert.ToInt32(ConfigurationManager.AppSettings["ServerTimerInterval2"]);     
//serverConfig.AssemblyFolder = ConfigurationManager.AppSettings["AssemblyFolder"];
//serverConfig.AssemblyListPath = ConfigurationManager.AppSettings["AssemblyListPath"]; 

serverConfig.PollingOnce = Convert.ToBoolean(ConfigurationManager.AppSettings["PollingOnce"]);
var jobServer = new Shift.JobServer(serverConfig);

• Have all settings required by running jobs be retrievable in the Shift server app.config, web.config, or any reachable global storage directly from those jobs. If using a global storage, the Shift server and app must also be authorized to connect to the global storage, since the jobs will be running under the server process.

Example of possible settings in config file:

<connectionStrings>
    <!--
    <add name="ShiftDBConnection" connectionString="localhost:6379" providerName="Redis" />
    <add name="ShiftDBConnection" connectionString="mongodb://localhost" providerName="MongoDB" />
    <add name="ShiftDBConnection" connectionString="https://localhost:8081/" providerName="DocumentDB" />
    -->
    <add name="ShiftDBConnection" connectionString="Data Source=localhost\SQL2014;Initial Catalog=ShiftJobsDB;Integrated Security=SSPI;" providerName="System.Data.SqlClient" />
</connectionStrings>

<appSettings>
    <!-- Shift Server config -->
    <add key="ApplicationID" value="Demo.MVC" />
    <add key="MaxRunnableJobs" value="10" />
    <add key="ShiftWorkers" value="2" />
    <!--
    <add key="ShiftPID" value="4d9dd3d371804165b9a5783051b8debe" />
    -->

    <add key="AssemblyFolder" value="client-assemblies\" />
    <!-- <add key="AssemblyListPath" value="client-assemblies\assemblylist.txt" /> -->

    <!-- 
    <add key="StorageMode" value="mongo" />
    <add key="StorageMode" value="documentdb" />
    <add key="DocumentDBAuthKey" value="**************" />
    <add key="StorageMode" value="redis" />
    -->
    <add key="StorageMode" value="mssql" />

    <!-- Set to 0 or low 1 sec for StorageMode = redis-->
    <add key="ProgressDBInterval" value="00:10:00" />

    <add key="ForceStopServer" value="true" />
    <add key="StopServerDelay" value="5000" />

    <!--
    <add key="AutoDeletePeriod" value="120" />
    <add key="ServerTimerInterval" value="5000" />
    <add key="ServerTimerInterval2" value="10000" />
    <add key="ShiftEncryptionParametersKey" value="[OPTIONAL_ENCRYPTIONKEY]" /> 
    <add key="PollingOnce" value="true" />
    -->
</appSettings>

Configuration

Configuration	Description
StorageMode	Required. Either a Redis server: 'redis', MongoDB "mongo", or Microsoft SQL server: 'mssql'
DBConnectionString	Required. Storage connection string.
DBAuthKey	DocumentDB authentication key, required if storage set to DocumentDB.
ProcessID	Default is auto generated. The unique identification of the Shift server process that's running the jobs.
UseCache (obsolete)	Removed in v1.0.8.1. Default is false. If true then the CacheConfigurationString is required.
CacheConfigurationString (obsolete)	Removed in v1.0.8.1. The cache configuration, required if UseCache is true.
AssemblyFolder	The assemblies folder location required for running jobs.
AssemblyListPath	The text file path and name containing the list of assemblies required for jobs.
MaxRunnableJobs	Default to 100. The maximum jobs that can run per Shift server.
Workers	Default to 1. Number of generated workers per Shift server.
EncryptionKey	Key to encrypt the serialized jobs. Must use the same encryption key as the Shift client that add jobs.
ProgressDBInterval	Default to 10 seconds. The main storage interval between progress update from running jobs.
ServerTimerInterval	Default to 5 seconds. The polling interval when Shift server checks job queues for runnable jobs.
ServerTimerInterval2	Default to 10 seconds. The polling interval when Shift server runs internal clean up.
AutoDeletePeriod	Default is none/null. Auto delete jobs in # hours or older. Set to null to turn off auto deletion.
AutoDeleteStatus	Default to Completed. Auto delete only jobs with status defined in config. Set to null to ignore status.
PollingOnce	Default to false. Set the server polling to run only once if set to true. Useful for debugging and testing.
ForceStopServer	Default to false. Force the server to quit, don't wait for any running jobs to run to completion.
StopServerDelay	Default to 30000 ms. The server waits up to ### before it quits.

By default, when stopping a server, the Shift server will attempt to stop running jobs and wait until all running they are stopped or if not cancellable to finish to 100% completion. Setting the ForceStopServer to true allows the server application to quit without waiting, however this may leave some running jobs into a zombie state, where their status in storage is marked as running, but in reality, they're not.

Restarting the server with the same process ID and number of workers will generally clean up the zombie jobs and mark them having an error and failed. If restarting server does not work, then changing job's status directly in storage will be necessary.

The ProcessID setting is not required, when Shift server auto generates a process ID (a GUID without dashes), the generated process ID is saved in a text file named shift_pid.txt. This file is located in the base folder of the server application. The process ID in shift_pid.txt file will be re-used after a Shift server restarts. By using the same process ID, the server guarantees that zombie jobs will be marked with error status correctly. DO NOT delete the shift_pid.txt before ensuring that all jobs has been completed or stopped.

If AssemblyListPath and AssemblyFolder are set, only the AssemblyFolder setting will be used by Shift Server. Use AssemblyListPath to specify specific assembly files to load, AssemblyFolder will load all *.DLL in the folder, no recursive path loading is supported at this time. The AssemblyFolder and AssemblyListPath can be set to empty for server app with direct references to the required job assemblies. Please refer to Assembly Loading for more info.

The Shift server can automatically deletes older jobs. To turn it on, set AutoDeletePeriod in a set of hours, to turn it off, set the AutoDeletePeriod to empty or null.

By default only completed jobs are deleted. To auto delete older jobs with no regard to their status, set the AutoDeleteStatus list to an empty list. To delete jobs with other statuses, set AutoDeleteStatus with multiple JobStatus, null is also a valid status, null value is equivalent to job is ready to run.

config.AutoDeleteStatus = new List<JobStatus?> { JobStatus.Stopped, JobStatus.Completed }; //Auto delete only jobs that had Stopped or Completed 

Paused jobs can continue only when the running process is kept alive by the server, any disruption to the server process will render the paused jobs into zombie jobs and the server will change those jobs to an Error state. To resolve this issue, the Shift client must reset and restart affected jobs.

Clean Up

The server will attempt to synchronize and clean up the running jobs with it's data in storage. The clean up method will:

• Mark jobs to Error state when the jobs are marked as running or paused but no corresponding processes are found. This typically occurs when the server goes down abruptly.
• Attempt to stop all jobs processes that's still running but do not exists anymore in storage.
• Remove any non running jobs from server memory to conserve resources.