From: Aaron Schulz <aschulz@wikimedia.org>
Date: Tue, 23 Oct 2012 18:32:46 +0000 (-0700)
Subject: [JobQueue] README file for job queue classes.
X-Git-Tag: 1.31.0-rc.0~21882
X-Git-Url: http://git.cyclocoop.org/%24action?a=commitdiff_plain;h=63bba0b35634362e212278bc731ad84e3b74f314;p=lhc%2Fweb%2Fwiklou.git

[JobQueue] README file for job queue classes.

Change-Id: Ib3e3eb4905511ba4d5fdfdf4f32279a6b6c1ac51
---

diff --git a/includes/job/README b/includes/job/README
new file mode 100644
index 0000000000..57c92e9270
--- /dev/null
+++ b/includes/job/README
@@ -0,0 +1,58 @@
+/*!
+\ingroup JobQueue
+\page jobqueue_design Job queue design
+
+Notes on the Job queuing system architecture.
+
+\section intro Introduction
+
+The data model consist of the following main components:
+
+* The Job object represents a particular deferred task that happens in the
+  background. All jobs subclass the Job object and put the main logic in the
+  function called run().
+* The JobQueue object represents a particular queue of jobs of a certain type.
+  For example there may be a queue for email jobs and a queue for squid purge
+  jobs.
+
+Each job type has its own queue and is associated to a storage medium. One
+queue might save its jobs in redis while another one uses would use a database.
+
+Storage medium are defined in a queue class. Before using it, you must
+define in $wgJobTypeConf a mapping of the job type to a queue class.
+
+The factory class JobQueueGroup provides helper functions:
+- getting the queue for a given job
+- route new job insertions to the proper queue
+
+The following queue classes are available:
+* JobQueueDB (stores jobs in the `job` table in a database)
+
+All queue classes support some basic operations (though some may be no-ops):
+* enqueueing a batch of jobs
+* dequeueing a single job
+* acknowledging a job is completed
+* checking if the queue is empty
+
+Some queue classes (like JobQueueDB) may dequeue jobs in random order while other
+queues might dequeue jobs in exact FIFO order. Callers should thus not assume jobs
+are executed in FIFO order.
+
+Also note that not all queue classes will have the same reliability guarantees.
+In-memory queues may lose data when restarted depending on snapshot and journal
+settings (including journal fsync() frequency).  Some queue types may totally remove
+jobs when dequeued while leaving the ack() function as a no-op; if a job is
+dequeued by a job runner, which crashes before completion, the job will be
+lost. Some jobs, like purging squid caches after a template change, may not
+require durable queues, whereas other jobs might be more important.
+
+Callers should also try to make jobs maintain correctness when executed twice.
+This is useful for queues that actually implement ack(), since they may recycle
+dequeued but un-acknowledged jobs back into the queue to be attempted again. If
+a runner dequeues a job, runs it, but then crashes before calling ack(), the
+job may be returned to the queue and run a second time. Jobs like cache purging can
+happen several times without any correctness problems. However, a pathological case
+would be if a bug causes the problem to systematically keep repeating. For example,
+a job may always throw a DB error at the end of run(). This problem is trickier to
+solve and more obnoxious for things like email jobs, for example. For such jobs,
+it might be useful to use a queue that does not retry jobs.