Fix message cache expiry semantics
* Use the stale message cache while the new one is being generated
* Revert I811755d4 (make message cache load failure fatal). This
escalated several very plausible temporary site issues from barely
noticeable to complete downtime -- for example, memcached being down
on a site with only one memcached server.
* Remove $wgLocalMessageCacheSerialized, it's always been pointless
* Clarify a couple of comments.
* Increased lock wait timeout to 30s
* Make lock() fail immediately on memcached connection refused
Tests done:
* With local cache enabled: normal cold refill; refill local from
global cache; use stale local cache during remote refill; use stale
global cache during remote refill; cold cache wait for remote refill;
saveToCaches() failure; memcached connection refused.
* With local cache disabled: saveToCaches() failure; cache disabled due
to "error" status key; memcached connection refused.
Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is
not the best idea since it inevitably leads to a cache stampede. Dealing
with the stampede by waiting for a lock is not ideal, even if it were
implemented properly, since it's not necessary to deliver perfectly
fresh message cache data to all clients.
This is especially obvious when you note that barring bugs, expiry and
regeneration always gives you back the exact same data, because we have
incremental updates (MessageCache::replace()). Keeping all clients
waiting for 10s just to give them the data they have already is pretty
pointless.
So, continue to serve the site from the stale message cache while the
new one is being generated.
One caveat: if local caching enabled, when the message cache becomes
stale, a sudden spike in network bandwidth may result due to the full
array (also typically stale) being fetched from the shared cache.
Bug: 43516
Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b