Workaround for bug 28146: running out of memory during Unicode validation/normalizati...

This provisional workaround runs a page at a time through UtfNormal::cleanUp() instead of running the entire file's dumped text at once. This avoids exploding memory too much during the preg_match_all() used to divide up ASCII and non-ASCII runs for validation, which is very wasteful for long texts in Latin languages with many mixed-in non-ASCII characters (like French and German text).
Won't fix legit cases of huge texts, such as realllllllllly long page text, which would still be subject to getting run through at web input time in a giant chunk.

diff --git a/includes/DjVuImage.php b/includes/DjVuImage.php
index 7a8b5e4..f2effcb 100644 (file)
--- a/includes/DjVuImage.php
+++ b/includes/DjVuImage.php
@@ -254,8 +254,7 @@ class DjVuImage {
                        $txt = wfShellExec( $cmd, $retval );
                        wfProfileOut( 'djvutxt' );
                        if( $retval == 0) {
-                               # Get rid of invalid UTF-8, strip control characters
-                               $txt = UtfNormal::cleanUp( $txt );
+                               # Strip some control characters
                                $txt = preg_replace( "/[\013\035\037]/", "", $txt );
                                $reg = <<<EOR
                                        /\(page\s[\d-]*\s[\d-]*\s[\d-]*\s[\d-]*\s*"
@@ -279,7 +278,8 @@ EOR;
        }
 
        function pageTextCallback( $matches ) {
-               return '<PAGE value="' . htmlspecialchars( $matches[1] ) . '" />';
+               # Get rid of invalid UTF-8, strip control characters
+               return '<PAGE value="' . htmlspecialchars( UtfNormal::cleanUp( $matches[1] ) ) . '" />';
        }
 
        /**