Rewrite of Tokenizer. Not using regex any more.
authorJens Frank <jeluf@users.mediawiki.org>
Sat, 10 Apr 2004 07:53:52 +0000 (07:53 +0000)
committerJens Frank <jeluf@users.mediawiki.org>
Sat, 10 Apr 2004 07:53:52 +0000 (07:53 +0000)
commitf9164a40fb4afeee962be37cfe8f1166fc6547f7
tree764104343bd355ff4c9905fcdeb8e79583a43c2e
parentd0229d46bf78f38007608dc4f1d664225537e23e
Rewrite of Tokenizer. Not using regex any more.

TODO: prefixed links for Arabic wikipedia not implemented yet
      handling of French blanks not yet implemented
includes/Tokenizer.php