Regular Expression Unicode Blocks in IronPython and jython

One last thing that's available in jython and IronPython, but not in CPython regular expressions is Unicode Blocks.  Blocks are similar to Unicode Scripts, but do not correspond one to one with them.  Blocks, as the name implies, represent continuous sequences of Unicode code points.  This page, recommended to me by artisonian on twitter, has a good synopsis.

Where Unicode Blocks are most useful (where they correspond best with Unicode Scripts) is in the South Asian languages (India and vicinity).  Here is some code written for the detection of Bengali characters in a string in IronPython and jython.  The syntax is similar.

Iron Python

/bin/mono /home/carl/IronPython-2.0.3/ipy.exe                                 <
IronPython 2.0.3 (2.0.0.0) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> from System.Text import RegularExpressions as regex
>>> fle = open('bengalisnippet', 'r')
>>> linex = fle.readline()
>>> fle.close()
>>> rex = regex.Regex(r'\p{IsBengali}+')
>>> linex = linex.decode('utf-8')
>>> mtchx = rex.Match(linex)
>>> mtchx.ToString()
u'\u0995\u09bf\u099b\u09c1'
>>> mtchx.Success
True
>>>

jython

Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.util import regex
>>> rex = regex.Pattern.compile(r'\p{InBengali}+')
>>> fle = open('bengalisnippet', 'r')
>>> linex = fle.readline()
>>> linex = linex.decode('utf-8')
>>> mtchx = rex.matcher(linex)
>>> mtchx
java.util.regex.Matcher[pattern=\p{InBengali}+ region=0,5 lastmatch=]
>>> mtchx.find()
True
>>> mtchx.start()
0
>>> mtchx.end()
4
>>> linex
u'\u0995\u09bf\u099b\u09c1\n'
>>>

IronPython, unicode, and regular expressions

This is a quick follow on to my last post on jython.

The basic idea is that .NET has the capability to search for characters belonging to Unicode general categories (in this case Mn for non-spacing character).

IronPython 2.0.3 (2.0.0.0) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> import unicodedata
>>> from System.Text import RegularExpressions as regex
>>> nonspacingx = regex.Regex(r'\p{Mn}')
>>> ns = unichr(0x9C1)
>>> ns
u'\u09c1'
>>> nonspacingx.Match(ns)
<System.Text.RegularExpressions.Match object at 0x000000000000002B [?]>
>>> ns = u'a' + ns
>>> ns
u'a\u09c1'

>>> mtchx = nonspacingx.Match(ns)
<System.Text.RegularExpressions.Match object at 0x000000000000002C [?]>
>>> mtchx.ToString()
u'\u09c1'
>>> mtchx.Index
1
>>> mtchx.Length
1
>>> mtchx.Success
True
>>> 

Although the names are different, Java and .NET both provide a means of using general categories in regular expressions.  Match in .NET matches occurrences within the string, not just at the start.  Success is the boolean value indicating a match.

As an aside, the unicodedata module referenced in the jython post is available for IronPython.  It is not in the download for either IronPython or FePy, but is available as a separate download from the FePy site.                               

jython, regular expressions, and unicode

Jython enables access to Java's regular expression classes and methods.  One feature of Java's regular expression library that Python does not have is the ability to search on Unicode general categories (http://unicode.org/Public/UNIDATA/PropertyValueAliases.txt).  These are abbreviations:  Mn = non-spacing character, Lu = uppercase letter, etc.

Here is a quick example for Mn (non-spacing).

$ /usr/local/jdk-1.7.0/bin/java -jar jython.jar
Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.util import regex
>>> nonspacingx = regex.Pattern.compile(r'\p{Mn}')
>>> import unicodedata
>>> for charcode in range(0x900, 0xA00):
...     mtchx = nonspacingx.matcher(unichr(charcode))
...     if mtchx.matches():
...         print 'match at character %X, %s' % (charcode, unicodedata.name(unichr(charcode)))
...
match at character 901, DEVANAGARI SIGN CANDRABINDU
match at character 902, DEVANAGARI SIGN ANUSVARA
match at character 93C, DEVANAGARI SIGN NUKTA
match at character 941, DEVANAGARI VOWEL SIGN U
match at character 942, DEVANAGARI VOWEL SIGN UU
match at character 943, DEVANAGARI VOWEL SIGN VOCALIC R
match at character 944, DEVANAGARI VOWEL SIGN VOCALIC RR
match at character 945, DEVANAGARI VOWEL SIGN CANDRA E
match at character 946, DEVANAGARI VOWEL SIGN SHORT E
match at character 947, DEVANAGARI VOWEL SIGN E
match at character 948, DEVANAGARI VOWEL SIGN AI
match at character 94D, DEVANAGARI SIGN VIRAMA
match at character 951, DEVANAGARI STRESS SIGN UDATTA
match at character 952, DEVANAGARI STRESS SIGN ANUDATTA
match at character 953, DEVANAGARI GRAVE ACCENT
match at character 954, DEVANAGARI ACUTE ACCENT
match at character 962, DEVANAGARI VOWEL SIGN VOCALIC L
match at character 963, DEVANAGARI VOWEL SIGN VOCALIC LL
match at character 981, BENGALI SIGN CANDRABINDU
match at character 9BC, BENGALI SIGN NUKTA
match at character 9C1, BENGALI VOWEL SIGN U
match at character 9C2, BENGALI VOWEL SIGN UU
match at character 9C3, BENGALI VOWEL SIGN VOCALIC R
match at character 9C4, BENGALI VOWEL SIGN VOCALIC RR
match at character 9CD, BENGALI SIGN VIRAMA
match at character 9E2, BENGALI VOWEL SIGN VOCALIC L
match at character 9E3, BENGALI VOWEL SIGN VOCALIC LL
>>> 

That's nice, but how about a less contrived example.  I got a Bengali word off one of the links on the BengaliLanguage page on the Python Wiki.  The word is saved to a file bengalisnippet.

All I have to do is open the file, get the line and let my regex rip, right?

>>> fle = open('bengalisnippet', 'r')
>>> linex = fle.readline()
>>> linex = linex.decode('utf-8')
>>> mtchx = nonspacingx.matcher(linex)
>>> mtchx.matches()
False
>>>

Um, no.

Let's investigate and try this again.

>>> linex

u'\u0995\u09bf\u099b\u09c1\n'
>>> unicodedata.category(linex[0])
'Lo'
>>> unicodedata.category(linex[1])
'Mc'
>>> unicodedata.category(linex[2])
'Lo'
>>> unicodedata.category(linex[3])
'Mn'

OK, the character we're looking for is the last one (except for the return character).

>>> mtchx.find()
True
>>> mtchx.start()
3
>>> mtchx.end()
4
>>>

I was using the wrong method (matches).  find is analogous to search in Python.

For me the utility of this is being able to determine if characters are rendering correctly.  I can locate the trouble spots in an unfamiliar language's script and investigate them (combining and non-spacing characters don't always show up correctly).

Java regular expressions are a bit more involved than Python's.  This is one case where the extra effort required may be worth the trouble.
                                       

java.lang.String.matches method

Recently I've been working on learning regular expressions.  Something about the Java implementation (in jython) I found curious.

Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.lang import String
>>> teststring = String('def hello():')
>>> teststring.matches(r'\s*def\s+\w*\(\):$')
True
>>>  

Python has the re.match and re.search methods.  C# has something similar.  This just seemed like a strange, less efficient construct (presumably the regular expression gets interpreted on the fly instead of compiled).  Go figure.

Second Javascript Attempt - the Zen of Python (again)

Last time, in my enthusiasm, I published some not ready for prime time html/JavaScript code.  Since then the W3C validator has helped me to see the error of my ways.  This is my second shot at making the first part of the Zen of Python magically appear in a web browser:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<HTML>
<HEAD>
<meta http-equiv="Content-type" content="text/html;charset=UTF-8">
<TITLE>
The Zen of Python, by Tim Peters
</TITLE>
<SCRIPT TYPE="text/javascript">
<!--

// function to put keys into Array
getkeysx = function(hashx) {
var keysy = [];
for (keyz in hashx) {
keysy.push(keyz);
}
return keysy;
}

addheaderx = function(headerid) {
var dx = document.createElement("DIV");
dx.id = 'div' + headerid;
var hx = document.createElement("H2");
hx.id = headerid;
hx.style.textAlign = 'center';
hx.style.fontFamily = 'sans-serif';
document.getElementById('bodyx').appendChild(dx);
document.getElementById('div' + headerid).appendChild(hx);
document.getElementById(headerid).innerHTML = '';
}

var timex = new Date();
var objx = {timetowork:timex,
number1:'Beautiful is better than ugly.',
number2:'Explicit is better than implicit.',
number3:'Simple is better than complex.',
number4:'Complex is better than complicated.',
number5:'Flat is better than nested.',
number6:'Sparse is better than dense.',
number7:'Readability counts.'};

var INTERVALX = 3000;

addheaders = function() {
var i = 1;
for (keyn in objx) {
addheaderx('header' + i);
i++;
}
}

var keysx = getkeysx(objx);
var keytracker = 0;
var colortracker = 0;
colorsx = ['red', 'green', 'blue', 'black', 'indigo',
'deeppink', 'darkslategray',
'darkmagenta', 'darkturquoise']

var counter = 1;
// function to write key-value pair to text box
writeprop = function() {
objx.timetowork = Date();
if (keytracker >= keysx.length) {
for (var j = 1; j < keysx.length + 1; j++) {
document.getElementById('header' + j).innerHTML = '';
}
keytracker = 0;
}
if (colortracker >= colorsx.length) {
colortracker = 0;
}
if (counter >= keysx.length + 1) {
counter = 1;
}
document.getElementById('header' + counter).innerHTML = objx[keysx[keytracker]];
document.getElementById('header' + counter).style.color = colorsx[colortracker];
setTimeout('writeprop()', INTERVALX);
keytracker++;
colortracker++;
counter++;
}

doall = function() {
addheaders();
writeprop();
}

// -->
</SCRIPT>
</HEAD>
<BODY ID = "bodyx" ONLOAD = "setTimeout('doall()', INTERVALX);">
<H1 ID = "zen" STYLE = "text-align:center;font-family:sans-serif">
THE ZEN OF PYTHON
</H1>
</BODY>
</HTML>

JavaScript attempt - the Zen of Python

I just completed a JavaScript course and couldn't resist messing with a web page (html file).  This rotates through the first part of the Zen of Python at five second intervals (warning - newbish code):

<HTML>
<DOCUMENT>
<HEAD>
<TITLE>
The Zen of Python, by Tim Peters
</TITLE>
<SCRIPT LANGUAGE="JavaScript">
<!--
var STYLEX = "border-width:0px;";
STYLEX += "border-style:solid;";
STYLEX += "font-family:sans-serif;";
STYLEX += "color:blue";

// function to put keys into Array
getkeysx = function(hashx) {
    var keysy = [];
    for (keyz in hashx) {
        keysy.push(keyz);
    }
    return keysy;
}

addbr = function() {
    var brx = document.createElement("BR");
    document.formx.appendChild(brx);
}

addx = function(idx) {
    addbr();
    var textx = document.createElement("INPUT");
    textx.type = ("TEXT");
    textx.value = "";
    textx.id = idx;
    textx.size = 50;
    textx.readonly = 'readonly';
    textx.style.borderWidth = '0px';
    textx.style.fontFamily = 'sans-serif';
    textx.style.fontSize = '2.75em';
    textx.style.color = 'blue';
    textx.style.textAlign = 'center';
    document.formx.appendChild(textx);
    addbr();
    addbr();
    addbr();
}

var timex = new Date();
var objx = {timetowork:timex,
number1:'Beautiful is better than ugly.',
number2:'Explicit is better than implicit.',
number3:'Simple is better than complex.',
number4:'Complex is better than complicated.',
number5:'Flat is better than nested.',
number6:'Sparse is better than dense.',
number7:'Readability counts.'};

addtextboxes = function() {
    var i = 1;
    for (keyn in objx) {
        addx('text' + i);
        i++;
    }
}

var keysx = getkeysx(objx);
var keytracker = 0;
var colortracker = 0;
colorsx = ['red', 'green', 'blue', 'black', 'indigo',
           'deeppink', 'darkslategray',
           'darkmagenta', 'darkturquoise']

var counter = 1;
// function to write key-value pair to text box
writeprop = function() {
    objx.timetowork = Date();
    if (keytracker >= keysx.length) {
        for (var j = 1; j < keysx.length + 1; j++) {
            document.formx['text' + j].value = '';
        }
        keytracker = 0;
    }
    if (colortracker >= colorsx.length) {
        colortracker = 0;
    }
    if (counter >= keysx.length + 1) {
        counter = 1;
    }
    document.formx['text' + counter].value = objx[keysx[keytracker]];
    document.formx['text' + counter].style.color = colorsx[colortracker];
    setTimeout('writeprop()', 5000);
    keytracker++;
    colortracker++;
    counter++;
}

doall = function() {
    addtextboxes();
    writeprop();
}

// -->
</SCRIPT>
</HEAD>
<BODY ONLOAD = "setTimeout('doall()', 3000);">
<FORM NAME = "formx">
</FORM>
</BODY>
</DOCUMENT>
</HTML>