pyright

pyeuclid, vector math, and polygon offset

2011-07-03T15:00:00.000-07:00

I had previously done a series of posts on polygon offset intended as a practical guide for accomplishing the task quickly. Some kind folks pointed out that I was making things considerably harder than necessary by using trigonometric functions when vector math would be easier and less error prone.

A coworker lent me his Java code that does polygon offset. I translated it into python (2.5) using the pyeuclid module:

import euclid as eu
import copy

OFFSET = 0.15

# coordinates
             # PT 1
MONASTERY = [(1.1, 0.75),
             # PT 2
             (1.2, 1.95),
             . . .
             # PT 21
             (1.1, 0.75)]

def scaleadd(origin, offset, vectorx):
    """
    From a vector representing the origin,
    a scalar offset, and a vector, returns
    a Vector3 object representing a point
    offset from the origin.

    (Multiply vectorx by offset and add to origin.)
    """
    multx = vectorx * offset
    return multx + origin

def getinsetpoint(pt1, pt2, pt3):
    """
    Given three points that form a corner (pt1, pt2, pt3),
    returns a point offset distance OFFSET to the right
    of the path formed by pt1-pt2-pt3.

    pt1, pt2, and pt3 are two tuples.

    Returns a Vector3 object.
    """
    origin = eu.Vector3(pt2[0], pt2[1], 0.0)
    v1 = eu.Vector3(pt1[0] - pt2[0],
                    pt1[1] - pt2[1], 0.0)
    v1.normalize()
    v2 = eu.Vector3(pt3[0] - pt2[0],
                    pt3[1] - pt2[1], 0.0)
    v2.normalize()
    v3 = copy.copy(v1)
    v1 = v1.cross(v2)
    v3 += v2
    if v1.z < 0.0:
        retval = scaleadd(origin, -OFFSET, v3)
    else:
        retval = scaleadd(origin, OFFSET, v3)
    return retval

polyinset = []
lenpolygon = len(MONASTERY)
i = 0
poly = MONASTERY
while i < lenpolygon - 2:
    polyinset.append(getinsetpoint(poly[i],
                 poly[i + 1], poly[i + 2]))
    i += 1
polyinset.append(getinsetpoint(poly[-2],
             poly[0], poly[1]))
polyinset.append(getinsetpoint(poly[0],
             poly[1], poly[2]))

The result:

2D geometry + infinite and zero slopes

2011-04-02T10:43:00.000-07:00

My last post dealt with a similar subject, but it didn't quite tell the whole story. No sooner had I posted it and attempted to implement it in a project, I ran into a problem. Granted, it was on a Java GUI in Java code (not a Python project) - but the concept is universal. When is a floating point number small enough that another mathematical operation on it will take it to zero or to the realm of infinity?

Python (I'm using CPython 2.5) is pretty forgiving in this sense.

>>> 1.0/1.0e-308
1e+308
>>> 1.0/1.0e-309
inf

There comes a point when even a representable number equals zero.

>>> 0.0 == 1.0e-10000
True

With some trigonometric functions from the math module:

>>> math.atan2(1.0, 0.0000000000000001)
1.5707963267948966
>>> 2 * _
3.1415926535897931
>>> _ == math.pi
True

>>> math.cos(1.0000000000000000001e-250)
1.0

The equation for the slope of a line in the Cartesian plane is (y2 - y1)/(x2 - x1) where y2, y1, x2, and x1 are the x and y coordinates of the enpoints of the line.

One way I've handled the infinite slope problem in the past is by setting a value as infinity:

# one million

>>> INFINITY = 1.0e+6
>>> slope = 1000000.0
>>> slope == INFINITY
True
This works sometimes, but it's not totally safe from error.

What I ended up doing this time was checking for zero or near zero values in the numerator and denominator of the slope equation.

NEARZERO = 1.0e-6
if math.abs(y2 - y1) < NEARZERO:
    # set y2 == y1 or vice versa
# and proceed with the calculations
    # for a zero slope
if math.abs(x2 - x1) < NEARZERO:
    # set x2 == x1 or vice versa
# and proceed with the calculations
    # for an infinite slope

This solved my problem in the sense that it gave reasonable results that served the problem domain well.

Lastly, you could use a similar method to check for duplicate points. If both numerator and denominator of the slope equation are near zero, the line segment is too small and the point should be done away with:

if (math.abs(y2 - y1) < NEARZERO and
    math.abs(x2 - x1) < NEARZERO):
    # duplicate point
    # delete it

Disclaimer: I am not a floating point math expert. There may be better solutions out there. These are a couple practical ones that have worked for me.

Polygon Offset Revisited

2011-03-27T13:36:00.000-07:00

I had previously written a post concerning polygon offset that was incomplete in its scope. Specifically, it dodged the problem of horizontal and vertical lines in polygons.

Since then I've had to confront this problem again. Shown below is the code I'm using for the more complete solution. It's a bit verbose, but it works.

import math

def calcoffsetpoint(pt1, pt2, offset):
    """
    Get a point offset from the line
    segment pt1-pt2 distance "offset".

    Return two tuple of coordinates.
    """
    theta = math.atan2(pt2[1] - pt1[1],
                       pt2[0] - pt1[0])
    theta += math.pi/2.0
    return (pt1[0] - math.cos(theta) * offset,
            pt1[1] - math.sin(theta) * offset)

def getoffsetintercept(pt1, pt2, m, offset):
    """
    From points pt1 and pt2 defining a line
    in the Cartesian plane, the slope of the
    line m, and an offset distance,
    calculates the y intercept of
    the new line offset from the original.
    """
    x, y = calcoffsetpoint(pt1, pt2, offset)
    return y - m * x

def getpt(pt1, pt2, pt3, offset):
    """
    Gets intersection point of the two
    lines defined by pt1, pt2, and pt3;
    offset is the distance to offset
    the point from the polygon.

    Valid for lines with slopes other
    than zero or infinity.

    Returns a two tuple of coordinates.
    """
    # get first offset intercept
    m = (pt2[1] - pt1[1])/(pt2[0] - pt1[0])
    boffset = getoffsetintercept(pt1, pt2, m, offset)
    # get second offset intercept
    mprime = (pt3[1] - pt2[1])/(pt3[0] - pt2[0])
    boffsetprime = getoffsetintercept(pt2, pt3, mprime, offset)
    # get intersection of two offset lines
    newx = (boffsetprime - boffset)/(m - mprime)
    newy = m * newx + boffset
    return newx, newy

def getslopeandintercept(pt1, pt2, offset):
    """
    Gets the slope and the intercept of the
    offset line.
    Result returned as a two tuple.
    """
    m = (pt2[1] - pt1[1])/(pt2[0] - pt1[0])
    b = getoffsetintercept(pt1, pt2, m, offset)
    return m, b

def getoffsetcornerpoint(pt1, pt2, pt3, offset):
    """
    Gets intersection point of the two
    lines defined by pt1, pt2, and pt3;
    offset is the distance to offset
    the point from the polygon.

    Returns a two tuple of coordinates.
    """
    # starting out with horizontal line
    if (pt2[1] - pt1[1]) == 0.0:
        ycoord = pt1[1] - math.cos(math.atan2(0.0, pt2[0] - pt1[0])) * offset
        # a vertical line follows
        if (pt3[0] - pt2[0]) == 0.0:
            xcoord = pt2[0] + math.sin(math.atan2(pt3[1] - pt2[1], 0.0)) * offset
        # a sloped line follows
        else:
            m, offsetintercept = getslopeandintercept(pt2, pt3, offset)
            # calculate for x with ycoord
            xcoord = (ycoord - offsetintercept)/m
    # starting out with a vertical line
    if (pt2[0] - pt1[0]) == 0.0:
        xcoord = pt1[0] + math.sin(math.atan2(pt2[1] - pt1[1], 0.0)) * offset
        # a horizontal line follows
        if (pt3[1] - pt2[1]) == 0.0:
            ycoord = pt2[1] - math.cos(math.atan2(0.0, pt3[0] - pt2[0])) * offset
        # a sloped line follows
        else:
            m, offsetintercept = getslopeandintercept(pt2, pt3, offset)
            # calculate for y with xcoord
            ycoord = m * xcoord + offsetintercept
    # starting out with sloped line
    if (pt2[1] - pt1[1]) != 0.0 and (pt2[0] - pt1[0]) != 0.0:
        # if second line is horizontal
        if (pt3[1] - pt2[1]) == 0.0:
            ycoord = pt2[1] - math.cos(math.atan2(0.0, pt3[0] - pt2[0])) * offset
            m, offsetintercept = getslopeandintercept(pt1, pt2, offset)
            # calculate for x with y coord
            xcoord = (ycoord - offsetintercept)/m
        # if second line is vertical
        elif (pt3[0] - pt2[0]) == 0.0:
            xcoord = pt2[0] + math.sin(math.atan2(pt3[1] - pt2[1], 0.0)) * offset
            m, offsetintercept = getslopeandintercept(pt1, pt2, offset)
            # solve for y with x coordinate
            ycoord = m * xcoord + offsetintercept
        # if both lines are sloped
        else:
            xcoord, ycoord = getpt(pt1, pt2, pt3, offset)
    return xcoord, ycoord

def offsetpolygon(polyx, offset):
    """
    Offsets a clockwise list of coordinates
    polyx distance offset to the inside of
    the polygon.
    Returns list of offset points.
    """
    polyy = []
    # need three points at a time
    for counter in range(0, len(polyx) - 3):
        # get first offset intercept
        pt = getoffsetcornerpoint(polyx[counter],
                   polyx[counter + 1],
                   polyx[counter + 2],
                   offset)
        # append new point to polyy
        polyy.append(pt)
    # last three points
    pt = getoffsetcornerpoint(polyx[-3], polyx[-2], polyx[-1], offset)
    polyy.append(pt)
    pt = getoffsetcornerpoint(polyx[-2], polyx[-1], polyx[0], offset)
    polyy.append(pt)
    pt = getoffsetcornerpoint(polyx[-1], polyx[0], polyx[1], offset)
    polyy.append(pt)
    return polyy

I used this polygon to test it.

This is a fairly simple geometric problem, but it can be confusing. Hopefully this will save someone the trouble of working through a solution.

Notes:

1) If you're using Python 2.x, feed your coordinates in as floats. This will ensure the absence of integer division, which often yields zeros, which in turn yields ZeroDivision errors.

2) The polygon fed to the offsetpolygon function should not have a closing point. The closing point will cause ZeroDivision errors. Instead, append the point before drawing the polygon in gnuplot or some other plotting tool.

jython + joda-time

2011-03-24T19:39:00.000-07:00

joda-time is a date-time library written in Java. It has advantages over Java's built in time utilities in terms of power and ease of use. It has some functionality that Python's own datetime library does not. Here is a sampling from within jython:

[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> # make sure the jar is in the CLASSPATH
>>> sys.path.append('/home/carl/Downloads/joda-time-1.6.2/joda-time-1.6.2.jar')
>>> from org.joda import time as joda
>>> rightnow = joda.DateTime()
>>> rightnow
2011-03-24T19:54:45.544-06:00
>>> rightnow.getZone()
America/Edmonton
>>> # I'm actually in New Mexico - close enough
>>> rightnow.dayOfWeek().getAsText()
u'Thursday'
>>> rightnow.monthOfYear().getAsText()
u'March'
>>> rightnow.getDayOfMonth()
24
>>> # days are one indexed along with months

>>> # a date from the past
>>> declofindependence = joda.DateTime(1776, 7, 4, 0, 0, 0, 0)

The Period object allows you to count down to dates in the future of count from dates in the past in terms of years, months, and days:

>>> americasage = joda.Period(declofindependence, rightnow)
>>> americasage.getYears()
234
>>> americasage.getMonths()
8
>>> americasage.getWeeks()
2
>>> americasage.getDays()
6

joda-time can handle dates far into the future as well as those in the far distant past:

>>> farfuture = joda.DateTime(10000, 1, 1, 1, 0, 0, 0)
>>> farfuture.getYear()
10000
>>> farfuture
10000-01-01T01:00:00.000-07:00
>>>

Like most things Java, joda-time can be a bit more verbose than its Python equivalent. Still, the ability to get dates beyond 9999 and some of the functionality may make it well worth the trouble.

WKT (Well Known Text) + JTS Topology + jython

2011-03-19T20:15:00.000-07:00

The last few posts have dealt with the JTS Topology library. This one will wrap up that topic with a method for converting JTS's Geometry objects to and from text, WKT.

WKT is a standard of the Open Geospatial Consortium (OGC). Objects expressed in WKT are not much different than their string representations in JTS. Below is a code example of working back and forth between JTS and WKT.

Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/home/carl/Downloads/jts/lib/jts-1.11.jar')
>>> from com.vividsolutions.jts.geom import GeometryFactory
>>> from com.vividsolutions.jts.geom import Coordinate
>>> from com.vividsolutions.jts.geom import LinearRing
>>> from com.vividsolutions.jts.io import WKTWriter
>>> from com.vividsolutions.jts.io import WKTReader
>>> coord1 = Coordinate(0, 0)
>>> coord2 = Coordinate(0, 1)
>>> coord3 = Coordinate(1, 1)
>>> coord4 = Coordinate(1, 0)
>>> coord5 = Coordinate(0, 0)
>>> coords = [coord1, coord2, coord3, coord4, coord5]
>>> from com.vividsolutions.jts.geom.impl import CoordinateArraySequence
>>> cas = CoordinateArraySequence(coords)
>>> gf = GeometryFactory()
>>> lr = LinearRing(cas, gf)
>>> asquare = gf.createPolygon(lr, None)
>>> f = open('wkttest.xml', 'w')
>>> wktwriter = WKTWriter()
>>> # this is just my own personal preference
>>> # it is inefficient for storage, but
>>> # sometimes easier on the eyes
>>> wktwriter.setMaxCoordinatesPerLine(1)
>>> text = wktwriter.writeFormatted(asquare)
>>> f.write(text)
>>> f.close()
>>> print text
POLYGON ((0 0,
0 1,
1 1,
1 0,
0 0))
>>> wktreader = WKTReader()
>>> f = open('wkttest.xml', 'r')
>>> text = f.read()
>>> asquare2 = wktreader.read(text)
>>> asquare2
POLYGON ((0 0, 0 1, 1 1, 1 0, 0 0))

It's fairly easy to store JTS Geometry objects in human readable (WKT) format and reuse them.

There is a good bit more to WKT than what's shown here. This was just a simple example for getting started.

WKB (Well Known Binary) is also available in JTS - this is used for efficient storage of Geometry objects, mainly in databases.

JTS also has a WKTFileReader class which can read in multiple geometries from a file.

Polygon Buffering with JTS Topology and jython

2011-03-18T19:45:00.000-07:00

Previously, I had done a post regarding polygon offset.  It turns out that JTS Topology has polygon buffering capability.  I wanted to try this out and compare it to my results from the polygon offset post.

As with most things with a library like JTS, the software does most of the work for you:

import sys
# path to jar into CLASSPATH
sys.path.append('/home/carl/Downloads/jts/lib/jts-1.11.jar')

from com.vividsolutions.jts.geom import Coordinate
from com.vividsolutions.jts.geom import GeometryFactory
# LinearRing is for the creation of the Polygon
from com.vividsolutions.jts.geom import LinearRing
# CoordinateSequenceArray is for the
# creation of the LinearRing
from com.vividsolutions.jts.geom.impl import CoordinateArraySequence
# CAP_ROUND is for the Polygon.buffer operation
from com.vividsolutions.jts.operation.buffer.BufferOp import CAP_ROUND

# coordinates
             # PT 1
MONASTERY = [(1.1, 0.75),
             # PT 2
             (1.2, 1.95),
             # PT 3
             (1.9, 1.96),
                .
                .
                .
             # PT 21
             (1.1, 0.75)]

gf = GeometryFactory()
coords = [Coordinate(*coord) for coord in MONASTERY]
cas = CoordinateArraySequence(coords)
lr = LinearRing(cas, gf)
polyx = gf.createPolygon(lr, None)

# offset of 0.15 inward
# rounded edges
# 10 points per quarter circle of rounded edges
polyinset = polyx.buffer(-0.15, 10, CAP_ROUND)

The result is shown below:

The rounded edges give an entirely different look to the shape. My version is shown below:

I'm glad I did the egg the way I did it (it looks more realistic). Nonetheless, the polygon buffering capability of JTS is a useful tool, particularly for calculating offsets and distances for scientific or geographic purposes.

Notes: Shapely is a Python package (CPython) with much of the same functionality as JTS.

Clipping Voronoi Polygons with JTS Topology and jython

2011-03-17T19:43:00.000-07:00

Following up on the last post on Voronoi diagrams and the JTS topology library, I will show clipping of the Voronoi polygons with another polygon (a polygonal bounding region) in this post.

The plot below is what we have currently, a Voronoi diagram (with points) clipped along a rectangular boundary:

Shown below is what I'd like to accomplish - clipping the red polygons with the blue surrounding one:

Here is the code, continued from the last post:

# because JTS can handle holes and "donuts", it
# has a LinearRing structure for constructing
# polygons
from com.vividsolutions.jts.geom import LinearRing
# the LinearRing requires a CoordinateArraySequence
# (it will not accept a list)
from com.vividsolutions.jts.geom.impl import CoordinateArraySequence

boundingcoords = [(10, 0), (2, 0),
                  (-2, 4), (2, 12),
                  (10, 6), (10, 0)]
boundingcoords = [Coordinate(*coord) for coord in boundingcoords]
casboundingcoords = CoordinateArraySequence(boundingcoords)
lr = LinearRing(casboundingcoords, geomfactx)
# None here fills in the spot where any inner rings
# would be in the polygon
boundingpoly = geomfactx.createPolygon(lr, None)

newpolygons = []
numpolygons = diagramx.getNumGeometries()
for numx in range(numpolygons):
    newpolygons.append(diagramx.getGeometryN(
        numx).intersection(boundingpoly))

And the result:

Note: it is possible to get more than one polygon as the result of each intersection. Through visual inspection I determined there would only be one polygon per intersection with the bounding area and structured my list (newpolygons) accordingly.

jython, JTS Topology, and Voronoi Diagrams

2011-03-15T19:48:00.000-07:00

Last time I barely scratched the surface of the JTS library. As it turns out, through ignorance I made a few things harder than they needed to be. The GeometryFactory class is quite versatile and has a number of methods for creating objects from simple Coordinate objects. In the example below, I've managed to create a clipped Voronoi diagram without too much code or difficulty.

import sys
# this line will be different depending on where
# the jts jar is stored and may be unnecessary if
# the jar is in your CLASSPATH
sys.path.append('/home/carl/Downloads/jts/lib/jts-1.11.jar')

from com.vividsolutions.jts.geom import Coordinate
from com.vividsolutions.jts.geom import GeometryFactory
from com.vividsolutions.jts.triangulate import VoronoiDiagramBuilder
from com.vividsolutions.jts.geom import Envelope

# first some coordinates
coord1 = Coordinate(5, 7)
coord2 = Coordinate(3, 6)
coord3 = Coordinate(1, 4)
coord4 = Coordinate(2, 5)
coord5 = Coordinate(4, 2)
coord6 = Coordinate(7, 2)
coord7 = Coordinate(5, 5)
coord8 = Coordinate(2, 6)
coord5 = Coordinate(4, 2)
coords = [coord1, coord2,
          coord3, coord4,
          coord5, coord6,
          coord7, coord8]

# the handy GeometryFactory
geomfactx = GeometryFactory()

# a MultiPoint Geometry object
mp = geomfactx.createMultiPoint(coords)

# the diagram builder class
vdb = VoronoiDiagramBuilder()

# it would be nice to clip the
# diagram with a rectangle;
# this is what the Envelope
# object is for
lowerleft = Coordinate(0, 0)
upperright = Coordinate(8, 8)
env = Envelope(lowerleft, upperright)
vdb.setClipEnvelope(env)

# load our sites (points) from the
# MultiPoint object
vdb.setSites(mp)

# this is where the builder object clips the
# the polygons with the Envelope rectangle
diagramx = vdb.getDiagram(geomfactx)

print 'Number of polygons = %d' % diagramx.getNumGeometries()
polygonx = diagramx.getGeometryN(0)
print 'One polygon:'
for coordsetx in polygonx.getCoordinates():
    print coordsetx

Output:

Number of polygons = 8
One polygon:
(-5.0, -4.0, NaN)
(-5.0, 8.25, NaN)
(0.5, 5.5, NaN)
(2.7, 3.3, NaN)
(-2.166666666666666, -4.0, NaN)
(-5.0, -4.0, NaN)

There's more that can be done here (for instance, clipping the polygons with a non-rectangular shape). As was true of the last post, this is just a small fraction of what the JTS Topology library can do.

jython + the JTS geometry library

2011-03-13T18:52:00.000-07:00

My favorite library for dealing with two dimensional points, lines, and polygons is Polygon by Jörg Rädler.  In a Java/jython environment, though, that's not available.  This led me to try my hand at JTS.  

Greg Corradini has a post that highlights JTS use with jython and geoscript.  His overview was very helpful, but I wanted to get to know JTS more thoroughly.

The first thing I came up against when using JTS was the documentation.  It's all javadocs.  This isn't necessarily a bad thing, but, along with the library's java-centric design, it takes a little getting used to.  If concepts like abstract class, iterface, extending, and implementing don't immediately ring a bell to the user, use of the javadocs will make them clear over time.

That shouldn't discourage us, though.  Let's fire up the interpreter and try something simple.

Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/home/carl/Downloads/jts/lib/jts-1.11.jar')

I usually store things I'm testing out in my Downloads folder (I'm using KDE on OpenBSD).  This may not be the best practice.  In any case, to import from jts I need access to the jar in my sys.path.

Making a Point object in JTS in a bit complicated.  There are Coordinate objects and there are Point objects.  The Coordinate is a more basic class with less functionality than the Point class.  Point inherits from the abstract class Geometry.  This leads it to have a number of methods, some meaningless, like convexHull.  Shown below is the process required to make a Point object.

>>> from com.vividsolutions.jts.geom import Point
>>> from com.vividsolutions.jts.geom import GeometryFactory
>>> from com.vividsolutions.jts.geom.impl import CoordinateArraySequence
>>> # get CoordinateArraySequence of size one
>>> cas = CoordinateArraySequence(1)
>>> factoryx = GeometryFactory()
>>> ptx = Point(cas, factoryx)

We have a Point object.  A couple explanatory notes:

 1) the documentation for the default GeometryFactory reads thus:  "Constructs a GeometryFactory that generates Geometries having a floating  PrecisionModel and a spatial-reference ID of 0."  For what I'm doing this is fine.

2) The preferred constructor for the Point object takes a CoordinateArraySequence of length one instead of a single Coordinate object.  This is analogous to a character in Python being a string of length one.

We don't know yet what our values are for our Point's coordinates.  Let's have a look.

>>> ptx
POINT (0 0)

OK, zero-zero.  Let's assign something to those coordinates away from the origin.

>>> ptx.getCoordinates()[0].x = 44
>>> ptx.getCoordinates()[0].y = 22
>>> ptx
POINT (44 22)

Even though the Point has only one coordinate, we still need to specify that with the index 0.

Well, that was a lot of work for just one point.  Let's try something more efficient and fun.

>>> cas2 = CoordinateArraySequence(6)
>>> somecoordinates = [(1, 5), (7, 14), (22, 44), (36, 12), (19, 1), (12, 4)]
>>> for numx in range(len(somecoordinates)):
...     cas2.getCoordinate(numx).x = somecoordinates[numx][0]
...     cas2.getCoordinate(numx).y = somecoordinates[numx][1]
...
>>> cas2
((1.0, 5.0, NaN), (7.0, 14.0, NaN), (22.0, 44.0, NaN), (36.0, 12.0, NaN), (19.0, 1.0, NaN), (12.0, 4.0, NaN))
>>> from com.vividsolutions.jts.geom import LineString
>>> ls = LineString(cas2, factoryx)
>>> ch = ls.convexHull()
>>> ch
POLYGON ((19 1, 1 5, 22 44, 36 12, 19 1))
>>> 

LineString is one of the classes that can be instantiated from a CoordinateArraySequence.  I went with that mainly for ease of use.
 
There is a great deal more to JTS.  This post was mainly for getting started with the library.

jython + Google Guava's base.CaseFormat

2011-03-02T19:51:00.000-08:00

I was trying to research Google Collections for use in jython and stumbled upon Solid Craft's blog and his description of Guava base's CaseFormat utility. Where I work we're a Java shop, and use of camelCase infiltrates all code, even jython code. CaseFormat provides functionality for changing these variable names to a number of different formats.

$ /usr/local/jdk-1.7.0/bin/java -jar jython.jar
Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> # get downloaded jar into sys.path
>>> sys.path.append('/home/carl/Downloads/guava-r08/guava-r08.jar')
>>> from com.google.common.base import CaseFormat as cf
>>> dir(cf)
['LOWER_CAMEL', 'LOWER_HYPHEN', 'LOWER_UNDERSCORE', 'UPPER_CAMEL', . . .]
>>> varname = 'someVariable'
>>> cf.LOWER_CAMEL.to(cf.LOWER_UNDERSCORE, varname)
u'some_variable'
>>> cf.LOWER_CAMEL.to(cf.UPPER_UNDERSCORE, varname)
u'SOME_VARIABLE'
>>> cf.LOWER_CAMEL.to(cf.UPPER_CAMEL, varname)
u'SomeVariable'
>>>

For cycling through a file full of camel case or underscored variable names, even by hand in the interpreter, this could save some time and typing.

jython + java.lang.Character.UnicodeBlock

2011-02-28T13:48:00.000-08:00

In my post on Unicode Blocks in regular expressions, I mentioned there wasn't support for Unicode Blocks in CPython regular expressions. Unicode Blocks are contiguous (by number) sections of the Unicode tables with some commonality among the characters. There is another Java language feature that can be helpful for those interested in Unicode Blocks: the java.lang.Character.UnicodeBlock object. In jython:

Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0_22
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.lang.Character import UnicodeBlock
>>> UnicodeBlock.of(236)
LATIN_1_SUPPLEMENT
>>> UnicodeBlock.of('a')
BASIC_LATIN
>>> UnicodeBlock.of(738)
SPACING_MODIFIER_LETTERS
>>> UnicodeBlock.of(922)
GREEK
>>> UnicodeBlock.of(0xffee)
HALFWIDTH_AND_FULLWIDTH_FORMS

The "of" method can accept either a character or a Unicode numeric identifier as it's argument. It provides a shorthand method of finding out roughly where a character is in Unicode and what it might represent.

More SVG - trace from bitmap with Inkscape and color editing

2011-02-24T21:18:00.000-08:00

I'm attempting to convert some of the pysanky eggs I drew with POV-ray to SVG so that I can scale them freely. Also, once they're converted, I would like to be able to edit their colors in the same way I did with the Python logo in my last post.

To the best of my knowledge, Inkscape (potrace) is the most readily available tool for converting the bitmap images from POV-ray to SVG. There are a few things to look out for, but this is still really easy:

1) Remove all reflection and shading from the textures (colors) in the POV-ray scene. Work on getting even lighting for the whole scene. Generate the scene without antialiasing (jagged edges are fine, and even desired).

2) Follow the directions here for tracing the SVG drawing.

Now to use the standard library's xml.etree.ElementTree package to get at the colors. Before jumping in, I first removed any references to the old bmp image from POV-ray and checked on the three color values in the SVG file.

The three colors in the image are #fefefe (slightly off white), #000000 (black), and #00fd00 (light green).

# svgchangecolor.py

"""
Manipulation of colors in an SVG image.
"""

from xml.etree import ElementTree as ET

def drilldown(nodex, oldcolor, newcolor):
    """
    Walk node of element tree in search
    of oldcolor.

    Replace oldcolor with newcolor where
    oldcolor occurs.
    """
    childrenx = nodex.getchildren()
    if childrenx:
        for childx in childrenx:
            drilldown(childx, oldcolor, newcolor)
    else:
        itemsx = nodex.items()
        for itemx in itemsx:
            if itemx[1].find(oldcolor) > -1:
                oldcolorstr = itemx[1]
                newcolorstr = oldcolorstr.replace(oldcolor, newcolor)
                nodex.set(itemx[0], newcolorstr)

def changecolor(oldcolor, newcolor, filename):
    """
    Replaces oldcolor with newcolor in an
    SVG file named filename.

    All three arguments are strings.

    oldcolor and newcolor are hex strings
    in the format #xxxxxx.
    """
    # the colors are at the lowest level
    # in the XML (SVG) file
    # drill down until they are reached
    svgobj = ET.parse(filename)
    rootx = svgobj.getroot()
    drilldown(rootx, oldcolor, newcolor)
    svgobj.write(filename)

$ python2.5
Python 2.5.4 (r254:67916, Aug 9 2010, 08:57:51)
[GCC 4.2.1 20070719 ] on openbsd4
Type "help", "copyright", "credits" or "license" for more information.
>>> import svgchangecolor as svgcc
>>> svgcc.changecolor('#00fd00', '#ffff00', 'crossdroppullegg.svg')
>>> svgcc.changecolor('#000000', '#00aa00', 'crossdroppullegg.svg')
>>> svgcc.changecolor('#fefefe', '#000000', 'crossdroppullegg.svg')

>>>

There are only three elements with color in the file (the dark green one is doing all the "work" of filling in the pattern). The script is really overkill in this case. For a drawing with a number of separate paths with the same color, though, this could be very useful.

SVG, XML, and the Python logo

2011-02-21T15:04:00.000-08:00

Legal notice:  the terms of use for the Python logo restrict the alteration or bastardization of the logo, including its colors.  I have contacted the Python Software Foundation and gotten permission to publish this blog entry.

The logo restrictions I violate in this post, for purposes of demonstrating working with svg, are:

   1) changing the color of the logo
   2) removal of the trademark symbol

While people are encouraged to use the logo, they are also encouraged to contact the PSF trademarks committee regarding questions of alteration.  Thank you for your consideration in this matter.

As a means of becoming more familiar with SVG (scalable vector graphics) and XML, I wanted to see if I could change the colors of the Python logo in code.  Although this is neither encouraged nor permitted (see notice above), it has been done on occasion in the past; I wanted to see how.

Since SVG is a subset of XML, I'm going to use an XML tool from the standard library (xml.etree.ElementTree) to work with the SVG image.

First, I downloaded the logo from python.org. I'm using the Java based tool Batik/Squiggle to view the SVG images:

To simplify the SVG file I removed the gray parts (text, trademark character, and shadow).  This I did by hand by looking for any gray colors of the hex form #xyxyxy in the file.

Also, I changed the name of the file I was referencing inside the SVG file to pythonlogochanged.svg.  This is important, otherwise the SVG file will continue to read whatever file is named inside it.

 

There are now six main elements to the file:
    1) the two shapes in the logo
    2) the two gradients for color variation across each shape
    3) the two sets of colors for start and stop in the gradation (two yellows and two blues)

It's number 3) that we're interested in.

I started out trying to use minidom from the xml standard library, but found ElementTree to be better suited to the task.  This tutorial from Effbot was helpful.

$ python2.5
Python 2.5.4 (r254:67916, Aug  9 2010, 08:57:51)
[GCC 4.2.1 20070719 ] on openbsd4
Type "help", "copyright", "credits" or "license" for more information.
>>> # open file
...
>>> svgobj = ET.parse('pythonlogochanged.svg')
>>> # get toplevel element
...
>>> rootx = svgobj.getroot()

Now it is time to drill down into the element tree.  I walked through this myself previously, so you'll have to trust that I know where the colors are in the tree.

>>> elementsx = rootx.getchildren()
>>> defsx = elementsx[2]
>>> elementsx = defsx.getchildren()
>>> yellowelement = elementsx[0]
>>> blueelement = elementsx[1]

First we'll look at the yellow color definitions.

>>> yellowchildren = yellowelement.getchildren()
>>> yellowchildren[0].items()
[('style', 'stop-color:#ffd43b;stop-opacity:1'), ('id', 'stop4673'), ('offset', '0')]
>>> yellowchildren[1].items()
[('style', 'stop-color:#ffe873;stop-opacity:1'), ('id', 'stop4675'), ('offset', '1')]

The yellow colors across the gradient are #ffd43b and #ffe873.
Now we'll look at the blue colors.

>>> bluechildren = blueelement.getchildren()
>>> bluechildren[0].items()
[('style', 'stop-color:#5a9fd4;stop-opacity:1'), ('id', 'stop4691'), ('offset', '0')]
>>> bluechildren[1].items()
[('style', 'stop-color:#306998;stop-opacity:1'), ('id', 'stop4693'), ('offset', '1')]

#5a9fd4 and #306998 are the blue colors across the gradient.

Let's try taking the color gradients out first without changing the blue and yellow colors.

>>> bluechildren[1].set('style', 'stop-color:#5a9fd4:stop-opacity:1')
>>> yellowchildren[1].set('style', 'stop-color:#ffd43b;stop-opacity:1')

Now the result needs to be written to disk.

>>> svgobj.write('pythonlogochanged.svg')


Still blue and yellow, but significantly different without the gradient.

Now let's make a black silhouette.

>>> bluechildren[0].set('style', 'stop-color:#000000:stop-opacity:1')
>>> bluechildren[1].set('style', 'stop-color:#000000:stop-opacity:1')
>>> yellowchildren[0].set('style', 'stop-color:#000000;stop-opacity:1')
>>> yellowchildren[1].set('style', 'stop-color:#000000;stop-opacity:1')
>>> svgobj.write('pythonlogochanged.svg')


The hardest part of this excercise was finding the appropriate parts of the SVG file to edit.  ElementTree makes it easy from there.

Simple Polygon Offset

2011-02-16T19:42:00.000-08:00

I was working on another POV-ray pysanky egg design and had to work with offsetting polygons. Stack Overflow had a brief description of how this could be done. I chose to go with the simplest solution: drawing offset lines and connecting their intersection points.

Update: ΤΖΩΤΖΙΟΥ was kind enough to introduce me to the math.atan2 function, which correctly identifies the quadrant the angle theta falls in, and rewrote my verbose function(s) concisely:

def calcoffsetpoint(pt1, pt2, offset):
    theta = math.atan2(pt2[1] - pt1[1],
                       pt2[0] - pt1[0])
    theta += math.pi/2.0
    return (pt1[0] - math.cos(theta) * offset,
            pt1[1] - math.sin(theta) * offset)

getoffsetintercept gets the b in y = mx + b needed to calculate the new point:

def getoffsetintercept(pt1, pt2, m, offset):
    """
    From points pt1 and pt2 defining a line
    in the Cartesian plane, the slope of the
    line m, and an offset distance,
    calculates the y intercept of
    the new line offset from the original.
    """
    x, y = calcoffsetpoint(pt1, pt2, offset)
    return y - m * x

The function that gets a single point along the polygon:

def getpt(pt1, pt2, pt3, offset):
    """
    Gets intersection point of the two
    lines defined by pt1, pt2, and pt3;
    offset is the distance to offset
    the point from the polygon.
    """
    # get first offset intercept
    m = (pt2[1] - pt1[1])/(pt2[0] - pt1[0])
    boffset = getoffsetintercept(pt1, pt2, m, offset)
    # get second offset intercept
    mprime = (pt3[1] - pt2[1])/(pt3[0] - pt2[0])
    boffsetprime = getoffsetintercept(pt2, pt3, mprime, offset)
    # get intersection of two offset lines
    newx = (boffsetprime - boffset)/(m - mprime)
    newy = m * newx + boffset
    return newx, newy

Lastly, the function that works the polygon offset:

def offsetpolygon(polyx, offset):
    """
    Offsets a clockwise list of coordinates
    polyx distance offset to the inside of
    the polygon.
    Returns list of offset points.
    """
    polyy = []
    # need three points at a time
    for counter in range(0, len(polyx) - 3):
        # get first offset intercept
        pt = getpt(polyx[counter],
                   polyx[counter + 1],
                   polyx[counter + 2],
                   offset)
        # append new point to polyy
        polyy.append(pt)
    # last three points
    pt = getpt(polyx[-3], polyx[-2], polyx[-1], offset)
    polyy.append(pt)
    pt = getpt(polyx[-2], polyx[-1], polyx[0], offset)
    polyy.append(pt)
    pt = getpt(polyx[-1], polyx[0], polyx[1], offset)
    polyy.append(pt)
    return polyy

Shown below is the shape I was trying to offset.

As the "Simple" in the entry's title suggests, this is a best case scenario:

1) no degenerate polygons inside or outside
2) no zero or infinite slopes

Actually, I should have some zero slopes, but I cheated and changed the coordinates. The differences won't be seen by the naked eye, but are big enough for the computer to handle them.

Here is the egg I was working on; it's a Hutsul design from the Sixty Score of Easter Eggs book:

Thanks for having a look.

jython copy versus clone

2011-02-08T14:42:00.000-08:00

We're a Java shop at work. We do a lot of passing of Java objects back and forth through the jython API to our application.

The other day I had the need to copy a Java object. It was a flat structure and my instinct was to use the object's clone method, which I did. This got me thinking about the use of clone versus the copy module in jython and I gave it a look:

Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0_22
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.util import ArrayList
>>> a = ArrayList()
>>> a
[]
>>> a.add(1)
True
>>> a.add([2, 3])
True
>>> import copy
>>> b = a.clone()
>>> b
[1, [2, 3]]
>>> b[1]
[2, 3]
>>> b[1][1]
3
>>> b[1][1] = 22
>>> b
[1, [2, 22]]
>>> a
[1, [2, 22]]
>>> b = copy.deepcopy(a)
>>> b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: Automatic proxy initialization should only occur on proxy classes

OK, clone works similar to copy.copy (shallow copy). Not sure what's going on with deepcopy, but it doesn't appear to be available in the case of an ArrayList object.

Update: as noted in the comments, the copy behavior on Java objects is a known bug and is scheduled to be fixed in jython version 2.5.2. In the meantime, clone() would appear to be a decent option.

Let's test the shallow copy capability of clone():

>>> a = ArrayList()
>>> a.add(1)
True
>>> a.add(2)
True
>>> a.add(3)
True
>>> a
[1, 2, 3]
>>> b = a.clone()
>>> b
[1, 2, 3]
>>> b[2] = 22
>>> b
[1, 2, 22]
>>> a
[1, 2, 3]

Perfect - it works just like copy.copy() would.

More POV-ray and Pysanky

2011-02-05T19:13:00.000-08:00

Continuing with the series of pysanky eggs I've been working on with Python and POV-ray - another week, another egg.  This one is a Lemko design.  The dots, I believe, represent stars.

Getting the crown and base was relatively easy, as it just required the correct positioning of a single stroke, then rotating it:

def makecrown(baseobj):
    crownstrokes = [baseobj]
    theta = STEP
    while theta < 360.0:
        crownstrokes.append(pov.Object(baseobj,
                            rotate = (0.0, theta, 0.0)))
        theta += STEP
    return pov.Union(*crownstrokes)

In this case, step = 360.0/24.0.

Likewise, I have a function for the dots that made them easier to place:

def movedot(basedot, xtheta, ytheta):
    dot = pov.Object(basedot, rotate = (xtheta, ytheta, 0.0))
    return pov.Object(dot, translate = (0.0, 1.0, 0.0))

Ultimately, I'd like to get enough egg designs coded to put together some sort of scene in POV-ray dedicated to the eggs. The following two scenes show what I have so far:

If I can identify a few more attractive, computationally friendly designs, I should be able to come up with enough eggs for a decent POV-ray egg scene. In the meantime, thanks for having a look.

POV-ray -another Pysanky Egg

2011-01-29T09:30:00.000-08:00

This is a continuation on a series of POV-ray posts all focused on the Pysanky decoration of egg shapes.  An attempt to represent the drop pull method of wax application to the egg is the subject of this post.  To represent this I've borrowed some work on flower shapes from a previous post and added a simple methodology for extending the flower in a simple V shape lengthwise:

def cycleequation(step, start, stop, trigfunc, factorr, factortheta, startend):
    """
    Returns list of x, y coordinates that correspond
    to the equation of the form

    factorr * r * r = (trigfunc(factortheta * theta)) ** 2

    theta will be incremented by step.
    start is the first value of theta.
    stop is the last value of theta.

    Shape starts and ends at startend, preferably a
    coordinate along the POV-ray z axis in the
    direction negative of the other points.
    """
    coords = [startend]
    theta = start
    while theta < stop:
        coords.append(getcoords(theta, factorr, factortheta, trigfunc))
        theta += step
    coords.append(getcoords(stop, factorr, factortheta, trigfunc))
    coords.append(startend)
    return coords

The head of the shape is formed by the equation

r ** 2 = (math.cos(2.0 * theta)) ** 2

from math.pi/4.0 + 0.2 to 3 * math.pi/4.0 - 0.2

For a long drop pull shape, a start and end point of (0.0, -15.0) yields a shape like the one pictured below:

The head and the lengthwise extension are slightly slope discordant. I did this intentionally to try to get the effect of the real life dropping of hot wax on the egg and pulling away from the drop to make the V shape.

With a parameter of (0.0, -5.0) for the start-stop point, a shorter, stubby drop is formed:

Through a series of scaling and rotation operations, I was able to get the drop shapes positioned to decorate my egg:

It takes 41 different strokes to make this egg (the way I approached it). I fell into the temptation briefly of naming the strokes 'stroke1', 'stroke2', etc. This is bad. Descriptive names are good. Even with a deceptively simple design like this, it's impossible to figure out what's going on and mirror the strokes that can be mirrored without descriptive names like 'northeastcenter', 'northwestcenter', etc.

Once I get a chance to rework the code, I can shorten the names and eliminate repetitive code. This pattern recognition/code refactoring exercise should keep me busy for a bit. In the meantime, thanks for having a look.

jython and java.util.Properties

2011-01-15T15:40:00.000-08:00

The other day my boss gave me an assignment to create a static properties file with stringified numeric mappings of file lookups.  Turns out this is really easy in jython (if you're used to Python instead of Java).  Hardly any Java is required:

The file itself looks like this:

#put comment here
#Sat Jan 15 16:24:57 MST 2011
3=numberthree.py
2=numbertwo.py
1=numberone.py

The value strings are stored in a manner that has them returned as Jython Unicode strings.
My case is the simplest one, in that the properties file is static. To update it dynamically and store it for the next use (for example, to maintain state), you would need a writable file object.

Prisms and Trigonometric Shapes - POV-ray

2011-01-09T12:18:00.000-08:00

I was looking for a way to put a four petal flower shape on my Python-POV-ray generated egg.  As it turns out, this shape is a common geometric one called Pascal's Rose.

For expediency rather than efficiency's sake, I opted to generate the shape using individual x, y points (actually x, z in POV-ray):

def getcoords(theta, factorr, factortheta, trigfunc):
    """
    Returns two tuple of x and y values for
    the equation in the form

    factorr * r * r = trigfunc(factortheta * theta)
    """
    r = math.sqrt((trigfunc(factortheta * theta)) ** 2 / factorr)
    x = r * math.cos(theta)
    y = r * math.sin(theta)
    if abs(x) < ZEROLIMIT:
        x = 0.0
    if abs(y) < ZEROLIMIT:
        y = 0.0
    print x, y, theta
    return x, y

def cycleequation(step, start, stop, trigfunc, factorr, factortheta):
    """
    Returns list of x, y coordinates that correspond
    to the equation of the form

    factorr * r * r = trigfunc(factortheta * theta)

    theta will be incremented by step.
    start is the first value of theta.
    stop is the last value of theta.
    """
    coords = []
    theta = start
    while theta < stop:
        coords.append(getcoords(theta, factorr, factortheta, trigfunc))
        theta += step
    coords.append(getcoords(stop, factorr, factortheta, trigfunc))
    return coords

math.cos creates Pascal's Rose for this equation.  I had to add a Prism object to the POV-ray recipe I am using.

class Prism(Item):
  def __init__(self, base, top, numpts, *opts, **kwargs):
    Item.__init__(self,"prism",(base, top, numpts),opts,**kwargs)

For me it was easiest to just translate the points of the prism into POV-ray SDL language prior to feeding them to Python:

def translatetopovray(pts):
    """
    Returns string in povray format of list
    of points pts.
    """
    stringpoints = []
    for pt in pts:
        stringpoints.append('<%.8f, %.8f>' % pt)
    return string.join(stringpoints, ',')

This allowed for generation of a prism:

This is just a rough representation of what I wanted.  The scene I was creating required a two colored flower, the outside as gold and the inside as orange.  Further, the outside edge needed to appear to be of constant width, as one would achieve by applying wax to an egg with the pysanky drawing tool, the kistka.

To get this effect I used a rough offset algorithm - get the centerpoint of the line segment on the outside edge of the "flower", offset a set distance along the normal to the outside line segment, then connect all the inner points.

def gettan(pt1, pt2):

    """
    Returns the tangent of the directional
    angle formed by line segment pt1-pt2.
    """
    return (pt2[1] - pt1[1])/(pt2[0] - pt1[0])

def getmidpoint(pt1, pt2):
    """
    Get the midpoint of the line segment
    pt1-pt2.
    """
    x = pt1[0] + (pt2[0] - pt1[0])/2.0
    y = pt1[1] + (pt2[1] - pt1[1])/2.0
    return x, y

def calcoffsetpoint(pt1, pt2, offset):
    """
    Get point offset distance offset
    from the midpoint of line segment
    pt1-pt2 to the left.
    """
    midpoint = getmidpoint(pt1, pt2)
    # get normal
    # quadrant I
    if pt2[0] >= pt1[0] and pt2[1] >= pt1[1]:
        theta = math.atan(gettan(pt1, pt2)) + math.pi/2.0
        x = midpoint[0] + math.cos(theta) * offset
        y = midpoint[1] + math.sin(theta) * offset
    # quadrant II
    elif pt2[0] <= pt1[0] and pt2[1] >= pt1[1]:
        theta = math.atan(gettan(pt1, pt2)) + math.pi/2.0
        x = midpoint[0] - math.cos(theta) * offset
        y = midpoint[1] - math.sin(theta) * offset
    # quadrant III
    elif pt2[0] <= pt1[0] and pt2[1] <= pt1[1]:
        theta = 3.0 * math.pi/2.0 + math.atan(gettan(pt1, pt2))
        x = midpoint[0] + math.cos(theta) * offset
        y = midpoint[1] + math.sin(theta) * offset
    # quadrant IV
    elif pt2[0] >= pt1[0] and pt2[1] <= pt1[1]:
        theta = 3.0 * math.pi/2.0 + math.atan(gettan(pt1, pt2))
        x = midpoint[0] - math.cos(theta) * offset
        y = midpoint[1] - math.sin(theta) * offset
    else:
        print pt1, pt2
        print 'error - theta undefined'
        theta = 0.0
    return x, y

def getmiddle(step, start, stop, trigfunc, factorr, factortheta, offset):
    """
    Return list of points for middle of 'petal'.
    """
    startend = (0.0, 0.0)
    pts = [startend]
    # need to offset by fixed distance along
    # normal to each line segment
    # need to get pts along original flower
    # but not near 0.0, 0.0 where they will
    # overlap
    outsidepts = cycleequation(step, start, stop,
        trigfunc, factorr, factortheta)
    counter = 0
    maxcounter = len(outsidepts)
    while (counter + 1) < (maxcounter - 1):
        pts.append(calcoffsetpoint(outsidepts[counter],
            outsidepts[counter + 1], offset))
        counter += 1
    pts.append(startend)
    return pts

My analytical geometry is a bit rough, but this works well enough for what I was trying to accomplish.

Through some intersecting and positioning, I was able to get the effect I wanted. This egg design, according to the Sixty Score of Easter Eggs book by Elyjiw, is from the Odessa region of the Ukraine.

Thanks for having a look.

POV-ray + Python - Scene Creation

2010-12-26T11:42:00.000-08:00

Brief recap: this recipe from the Python cookbook webpage is the basis for an api for POV-ray from Python. I added an Object class to the api for more flexibility in manipulating pre-made POV-ray objects.

This final post in the POV-ray series deals with an attempt to create a scene. I want to draw some pysanky eggs on a board with pegs in it for the eggs to stand on.

For the wood texture of the board I've used T_Wood10 from the POV-ray woods.inc library. This requires the inclusion of the woods.inc library in the Python code:

file = pov.File('test5.pov', 'colors.inc', 'woods.inc')

test5.pov is my output file; colors.inc and woods.inc are my include files.

Because the api deals exclusively with text, the wood texture will have to be put in as a string literal complete with POV-ray's curly brackets:

testblock = pov.Box((-15, -6.50, -18), (15, -2.00, -3), 

    texture = '{T_Wood10}')

As for the scene itself:

This is good for illustrating the scene's design, but it's not very realistic. The eggs look like they're wrapped in cellophane. I tried introducing some focal blur:

cam = pov.Camera(location = (0, 5, -35), look_at = (0, 0, 2),
    focal_point = (0, 2, -10), aperture = 0.4, blur_samples = 20)

This looks more realistic, but blurry. It would probably serve a remote corner of a scene better rather than the main object.

At this point it's really the nuances of POV-ray scene creation that will make a difference. The purpose of this post and the last three was to show that you can do some fun things with POV-ray from Python.

Lastly, and just for fun, some wooden eggs (textures textures.inc/Cork, woods.inc T_Wood10 and T_Wood14):

Coding POV-ray in Python

2010-12-21T17:43:00.000-08:00

My past two posts were basically on the same topic as this one. The difference now is that I was able to write Python code as I would normally write it with functions, dictionaries, and loops and still come out with valid POV-ray code on the other end. It's about a third of the length of the equivalent POV-ray code and, for me at least, easier to understand and read.

Here is the last part of the code that resulted in the scene of the three eggs below:

file = pov.File('test4.pov', 'colors.inc')
cam = pov.Camera(location = (-2, 3, -12), look_at = (0, 0, 2))
sorokolines = makesorokolines()testblock = makeredwedges()
testblockred = pov.Object(testblock, RED)
testblock = makeblackwedges()
testblockblack = pov.Object(testblock, BLACK)
test = pov.Union(testblockred, testblockblack, sorokolines)
test1 = pov.Object(test, translate = (2, 2, 0))
test2 = pov.Object(test, translate = (-2, -2, 0))
test3 = pov.Object(test, translate = (-6, 2, 0))
file.write(cam, test1, test2, test3, light1, light2, light3, light4, light5)

Now that I've proven to myself that I can code this in Python, I'd like to try something more involved like putting the eggs on a surface with a texture.

Thanks for having a look.

More POV-ray

2010-12-13T18:52:00.000-08:00

Since last time I've had some success porting POV-ray code to Python code based on this recipe by Simon Burton.

To briefly recap, I'm trying to reproduce this egg with Python code:

Since my first attempt, this is how far I've gotten:

Basically, the lines; I've made them a bit thicker to emphasize them. I'm still working on getting the camera and look at values set to coincide with the main axes.

The pictures are all well and good, but what's pretty exciting is that I made the second one using the API from the recipe. Further, I was able to expand on the recipe by adding an Object class:

class Object(Item):
def __init__(self, *opts, **kwargs):
Item.__init__(self, "object", (), opts, **kwargs)

This was really simple; I just copied what Simon Burton had done with the other classes. Still, it opened up a lot of possibilities for twirling, flipping, and coloring elements once they're constructed. For example, the code for the dividing lines above is (I've skipped the egg shape code for brevity):

white = pov.Texture(pov.Pigment(color = (1.0, 1.0, 1.0)), 
    pov.Finish(phong = redphng, reflection = redrflct))

horizontaldividingline = pov.Box((-3, 2.15, -3), (3, 2.25, 3), white)
verticaldividingline = pov.Box((-0.05, -1, -3), (0.05, 7, 3), white)

# vectors for dividing lines
# down on y axis
movedown = (0, -2.2, 0)
# up on y axis
mvup = (0, 2.2, 0)
# flip right around z axis
flprt = (0, 0, -52.5)

# scale - same in all directions (2)
scal = (2, 2, 2)

# turn for display
trn15back = (0, -15, 0)

planes = [horizontaldividingline, verticaldividingline]

# vertical planes
EIGHTHTURN = 45
turn = 45
for counter in range(3):
    plane = pov.Object(verticaldividingline, rotate = (0, turn, 0))
    planes.append(plane)
    turn += EIGHTHTURN

# for dividing lines at angle to horizontal
QUARTERTURN = 90
flprtdivlinepre = pov.Object(horizontaldividingline,
    translate = movedown, rotate = flprt)
flprtdivline = pov.Object(flprtdivlinepre, translate = mvup)

planes.append(flprtdivline)
turn = 90
for counter in range(3):
    plane = pov.Object(flprtdivline, rotate = (0, turn, 0))
    planes.append(plane)
    turn += QUARTERTURN

sorokoplanes = pov.Union(*planes)

eggwhite = pov.Object(unionegg, white, scale = scal)

sorokolines = pov.Intersection(eggwhite, sorokoplanes)

sorokotest = pov.Object(sorokolines, translate = (0, -2.75, 0), rotate = trn15back)

The egg design is very symmetrical and lends itself to repetition.  I tried to use this to my advantage with the two loops.  The list unpacking also compresses the code a bit.

Next on the agenda is placing code in functions and classes.  This would allow for making multiple eggs of different colors with a single code call.

POV-ray

2010-12-05T11:11:00.000-08:00

I messed around with POV-ray a bit about five years ago and recently tried to resurrect some of that code.

There's a recipe for a POV-ray - Python API by Simon Burton out on ActiveState that I wanted to try.

Here is the shape I was trying to re-create with the Python API:

Literally, an Easter egg, a bit involved, but not overly complex. The egg shape is borrowed from Friedrich Lohmüller's POV-ray site.

There is a simple example in the API which I've slightly modified to make a partially lit sphere:

# renamed recipe as pypov
import pypov as pov

file = pov.File('test2.pov', 'colors.inc')
cam = pov.Camera(location = (0, 1, -5), look_at = (0, -0.5, 2))
sphere = pov.Sphere((0, 0, 0), 1.5, pov.Texture(pov.Pigment(color = 'Blue')))
light = pov.LightSource((2, 4, -3), color = 'White')
file.write(cam, sphere, light)

This, after its output is run through POV-ray, yields this:

It won't win any animation awards, but it's pretty nonetheless.

The code for Lohmüller's egg shape looks like this:

# renamed recipe as pypov
import pypov as pov

file = pov.File('test3.pov', 'colors.inc')
cam = pov.Camera(location = (0, 1, -5), look_at = (0, -0.5, 2))
sphereupper = pov.Sphere((0, 0, 0), 1.0, pov.Texture(pov.Pigment(color = 'Blue')), scale = (1, 1.55, 1))
slabupper = pov.Box((-1, -1.55, -1), (1, 0, 1))
diffupper = pov.Difference(sphereupper, slabupper)
spherelower = pov.Sphere((0, 0, 0), 1.0, pov.Texture(pov.Pigment(color = 'Blue')), scale = (1, 1.15, 1))
slablower = pov.Box((-1, 0, -1), (1, 1.15, 1))
difflower = pov.Difference(spherelower, slablower)
union = pov.Union(difflower, diffupper, translate = (0, 0.55, 0), scale = 1.0)
light = pov.LightSource((2, 4, -3), color = 'White')
light2 = pov.LightSource((-2, -4, -3), color = 'White')
file.write(cam, union, light, light2)

and the output looks like this:

That is about as far as I got with the Python API. The problems I was having were related to trying to shoehorn my POV-ray code into the API. I added an Object class for the purpose of assigning attributes to predefined shapes. The problem there is that you can't use the same keyword more than once (translate, then rotate, then translate again).

Going forward I plan to work with simpler shapes (merging two parts instead of 50 or so). Also, I'll need to leverage what the API offers against working within its limitations. It will not be a one to one code translation between POV-ray and Python.

Regular Expression Unicode Blocks in IronPython and jython

2010-10-25T17:01:00.000-07:00

One last thing that's available in jython and IronPython, but not in CPython regular expressions is Unicode Blocks.  Blocks are similar to Unicode Scripts, but do not correspond one to one with them.  Blocks, as the name implies, represent continuous sequences of Unicode code points.  This page, recommended to me by artisonian on twitter, has a good synopsis.

Where Unicode Blocks are most useful (where they correspond best with Unicode Scripts) is in the South Asian languages (India and vicinity).  Here is some code written for the detection of Bengali characters in a string in IronPython and jython.  The syntax is similar. 

Iron Python

/bin/mono /home/carl/IronPython-2.0.3/ipy.exe <
IronPython 2.0.3 (2.0.0.0) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> from System.Text import RegularExpressions as regex
>>> fle = open('bengalisnippet', 'r')
>>> linex = fle.readline()
>>> fle.close()
>>> rex = regex.Regex(r'\p{IsBengali}+')
>>> linex = linex.decode('utf-8')
>>> mtchx = rex.Match(linex)
>>> mtchx.ToString()
u'\u0995\u09bf\u099b\u09c1'
>>> mtchx.Success
True
>>>

jython

Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.util import regex
>>> rex = regex.Pattern.compile(r'\p{InBengali}+')
>>> fle = open('bengalisnippet', 'r')
>>> linex = fle.readline()
>>> linex = linex.decode('utf-8')
>>> mtchx = rex.matcher(linex)
>>> mtchx
java.util.regex.Matcher[pattern=\p{InBengali}+ region=0,5 lastmatch=]
>>> mtchx.find()
True
>>> mtchx.start()
0
>>> mtchx.end()
4
>>> linex
u'\u0995\u09bf\u099b\u09c1\n'
>>>

IronPython, unicode, and regular expressions

2010-10-23T11:55:00.000-07:00

This is a quick follow on to my last post on jython.

The basic idea is that .NET has the capability to search for characters belonging to Unicode general categories (in this case Mn for non-spacing character).

IronPython 2.0.3 (2.0.0.0) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
>>> import unicodedata
>>> from System.Text import RegularExpressions as regex
>>> nonspacingx = regex.Regex(r'\p{Mn}')
>>> ns = unichr(0x9C1)
>>> ns
u'\u09c1'
>>> nonspacingx.Match(ns)
<System.Text.RegularExpressions.Match object at 0x000000000000002B [?]>
>>> ns = u'a' + ns
>>> ns
u'a\u09c1'

>>> mtchx = nonspacingx.Match(ns)
<System.Text.RegularExpressions.Match object at 0x000000000000002C [?]>
>>> mtchx.ToString()
u'\u09c1'
>>> mtchx.Index
1
>>> mtchx.Length
1
>>> mtchx.Success
True
>>>

Although the names are different, Java and .NET both provide a means of using general categories in regular expressions. Match in .NET matches occurrences within the string, not just at the start. Success is the boolean value indicating a match.

As an aside, the unicodedata module referenced in the jython post is available for IronPython. It is not in the download for either IronPython or FePy, but is available as a separate download from the FePy site.

jython, regular expressions, and unicode

2010-10-23T09:01:00.000-07:00

Jython enables access to Java's regular expression classes and methods. One feature of Java's regular expression library that Python does not have is the ability to search on Unicode general categories (http://unicode.org/Public/UNIDATA/PropertyValueAliases.txt). These are abbreviations: Mn = non-spacing character, Lu = uppercase letter, etc.

Here is a quick example for Mn (non-spacing).

$ /usr/local/jdk-1.7.0/bin/java -jar jython.jar
Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.util import regex
>>> nonspacingx = regex.Pattern.compile(r'\p{Mn}')
>>> import unicodedata
>>> for charcode in range(0x900, 0xA00):
...     mtchx = nonspacingx.matcher(unichr(charcode))
...     if mtchx.matches():
...         print 'match at character %X, %s' % (charcode, unicodedata.name(unichr(charcode)))
...
match at character 901, DEVANAGARI SIGN CANDRABINDU
match at character 902, DEVANAGARI SIGN ANUSVARA
match at character 93C, DEVANAGARI SIGN NUKTA
match at character 941, DEVANAGARI VOWEL SIGN U
match at character 942, DEVANAGARI VOWEL SIGN UU
match at character 943, DEVANAGARI VOWEL SIGN VOCALIC R
match at character 944, DEVANAGARI VOWEL SIGN VOCALIC RR
match at character 945, DEVANAGARI VOWEL SIGN CANDRA E
match at character 946, DEVANAGARI VOWEL SIGN SHORT E
match at character 947, DEVANAGARI VOWEL SIGN E
match at character 948, DEVANAGARI VOWEL SIGN AI
match at character 94D, DEVANAGARI SIGN VIRAMA
match at character 951, DEVANAGARI STRESS SIGN UDATTA
match at character 952, DEVANAGARI STRESS SIGN ANUDATTA
match at character 953, DEVANAGARI GRAVE ACCENT
match at character 954, DEVANAGARI ACUTE ACCENT
match at character 962, DEVANAGARI VOWEL SIGN VOCALIC L
match at character 963, DEVANAGARI VOWEL SIGN VOCALIC LL
match at character 981, BENGALI SIGN CANDRABINDU
match at character 9BC, BENGALI SIGN NUKTA
match at character 9C1, BENGALI VOWEL SIGN U
match at character 9C2, BENGALI VOWEL SIGN UU
match at character 9C3, BENGALI VOWEL SIGN VOCALIC R
match at character 9C4, BENGALI VOWEL SIGN VOCALIC RR
match at character 9CD, BENGALI SIGN VIRAMA
match at character 9E2, BENGALI VOWEL SIGN VOCALIC L
match at character 9E3, BENGALI VOWEL SIGN VOCALIC LL
>>>

That's nice, but how about a less contrived example.  I got a Bengali word off one of the links on the BengaliLanguage page on the Python Wiki.  The word is saved to a file bengalisnippet.

All I have to do is open the file, get the line and let my regex rip, right?

>>> fle = open('bengalisnippet', 'r')
>>> linex = fle.readline()
>>> linex = linex.decode('utf-8')
>>> mtchx = nonspacingx.matcher(linex)
>>> mtchx.matches()
False
>>>

Um, no.

Let's investigate and try this again.

>>> linex

u'\u0995\u09bf\u099b\u09c1\n'
>>> unicodedata.category(linex[0])
'Lo'
>>> unicodedata.category(linex[1])
'Mc'
>>> unicodedata.category(linex[2])
'Lo'
>>> unicodedata.category(linex[3])
'Mn'

OK, the character we're looking for is the last one (except for the return character).

>>> mtchx.find()
True
>>> mtchx.start()
3
>>> mtchx.end()
4
>>>

I was using the wrong method (matches). find is analogous to search in Python.

For me the utility of this is being able to determine if characters are rendering correctly. I can locate the trouble spots in an unfamiliar language's script and investigate them (combining and non-spacing characters don't always show up correctly).

Java regular expressions are a bit more involved than Python's. This is one case where the extra effort required may be worth the trouble.

java.lang.String.matches method

2010-10-18T18:50:00.000-07:00

Recently I've been working on learning regular expressions.  Something about the Java implementation (in jython) I found curious.

Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.7.0-internal
Type "help", "copyright", "credits" or "license" for more information.
>>> from java.lang import String
>>> teststring = String('def hello():')
>>> teststring.matches(r'\s*def\s+\w*\(\):$')
True
>>>  

Python has the re.match and re.search methods. C# has something similar. This just seemed like a strange, less efficient construct (presumably the regular expression gets interpreted on the fly instead of compiled). Go figure.

Second Javascript Attempt - the Zen of Python (again)

2010-10-03T16:32:00.000-07:00

Last time, in my enthusiasm, I published some not ready for prime time html/JavaScript code. Since then the W3C validator has helped me to see the error of my ways. This is my second shot at making the first part of the Zen of Python magically appear in a web browser:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<HTML>
<HEAD>
<meta http-equiv="Content-type" content="text/html;charset=UTF-8">
<TITLE>
The Zen of Python, by Tim Peters
</TITLE>
<SCRIPT TYPE="text/javascript">

</SCRIPT>
</HEAD>
<BODY ID = "bodyx" ONLOAD = "setTimeout('doall()', INTERVALX);">
<H1 ID = "zen" STYLE = "text-align:center;font-family:sans-serif">
THE ZEN OF PYTHON
</H1>
</BODY>
</HTML>

JavaScript attempt - the Zen of Python

2010-10-01T17:57:00.000-07:00

I just completed a JavaScript course and couldn't resist messing with a web page (html file). This rotates through the first part of the Zen of Python at five second intervals (warning - newbish code):

<HTML>
<DOCUMENT>
<HEAD>
<TITLE>
The Zen of Python, by Tim Peters
</TITLE>
<SCRIPT LANGUAGE="JavaScript">

</SCRIPT>
</HEAD>
<BODY ONLOAD = "setTimeout('doall()', 3000);">
<FORM NAME = "formx">
</FORM>
</BODY>
</DOCUMENT>
</HTML>

More RTL Python Editor

2010-08-23T12:44:00.000-07:00

Last time I tried to introduce the idea of an RTL Python editor for RTL languages like Arabic, Persian, Urdu, and Hebrew.

The idea is a little further along. I've gotten some Arabic and Urdu fonts installed. Also, the editor is capable of handling dictionaries, lists, tuples, and classes (there is code for brace type characters and an Arabic comma).

Here is a screenshot for a function that is all in Arabic except for the Python keywords (apologies to the Arabic speaking readership - the words are probably nonsense, as I just cut and pasted them randomly from the Python wiki's Arabic page):

The editor is written in Java. I'm working on writing the transfer to and from interpretable Python code in jython.

The approach I've taken is, if not a brute force one, definitely a forced one. Trying to mix and match left to right with bidirectional and right to left is tricky. To get around this I treat everything between whitespace and special characters (braces, colons, periods) as individual pieces of text. I further separate the text from whitespace and special characters through the use of the non-spacing, invisible Unicode character 200e. That character forces the editor back into left to right mode between words. Time will tell if this was a viable design decision.

For now it's one more step away from vaporware and towards releasable open source software.

Attempt at an RTL Editor for Python

2010-03-09T19:55:00.000-08:00

In previous posts, I've written about the possibilities offered by Python 3.1's Unicode identifier capability, as well as the new challenges posed when one tries to display them on screen.

As the final project for my Java course, I set about trying to create an editor that would allow the user to enter text right to left, but save it as valid interpretable (is that a word?) Python 3.x code:

There's still a good deal of work to be done on this, but this is a start. Hopefully I'll have something more robust next time.

Unicode Poster From Pycon 2010 Up

2010-02-24T18:16:00.000-08:00

I posted my poster to Slideshare in Open Office Presentation format. The file is about 13 or 14 megs in size. The embedded poster was in png format to preserve the shape of the foreign glyphs.

A number of people asked me at the poster session for sources. Here are the main ones:

1) Wikipedia - perhaps not the ultimate authority on all things, but a good place to research foreign languages and scripts.

2) the O'Reilly book Fonts and Encodings by Haralambous. If you know little about Unicode and fonts, this is the next best thing to Knuth.

3) the Python 3.1 interpreter and the unicodedata module. Once you get the basics of Unicode down, the unicodedata module has most of what you'll need.

4) Google searches and language promotion websites - laoconnection is a site that comes to mind. Most people are proud of their languages and culture and want to share them.

Thanks to everyone who stopped by the poster. That was fun.

FOSS Conference Economics

2010-02-23T09:21:00.000-08:00

Just got back from Pycon - great show.

I've had some time to reflect on how to make conference going affordable, and where my money goes. This year I was partially funded by my employer. I was quite grateful, as I wasn't expecting anything.

What has concerned me in the past is the amount of money put out on travel and hotels. If you're inside the US, Pycon(US) will see the biggest chunk of your money going to the hotel. This is where people (or at least me) say, "Hey, wait a second, all my monetary support for the Open Source Software movement is going to the hotel industry!" Not so fast - actually, although your money doesn't support FOSS directly, it does keep it from *losing* money. To secure a hotel/convention facility for more than 1000 people, there has to be a commitment on rooms. I've seen other devs stay at cheaper hotels for conferences - this is a good approach, if it's done out of necessity. I generally try to stay at the conference hotel in order to support the continued success of the conference - to make sure the conference doesn't lose money.

The travel argument goes roughly the same way - you can't have a conference if people don't show. Even though most of your money is going to the airlines (in the case of Pycon(US) for those outside the United States), your attendance is a plus.

Handling UnicodeEncodeError in the Console (Python 3.1)

2010-02-07T18:04:00.000-08:00

I've been working with a lot of different foreign scripts for the past six months or so. Ideally I like to work in the console where possible. An error that always comes up is the following:

[carl@pcbsd]/home/carl(139)% python3.1
Python 3.1.1 (r311:74480, Jan 17 2010, 23:15:26)
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> print('\u0400')
Traceback (most recent call last):
File "", line 1, in
UnicodeEncodeError: 'ascii' codec can't encode character '\u0400' in position 0: ordinal not in range(128)
>>>

After a while this can get pretty annoying. There's a number of ways to get around the problem. I don't know much about most of the languages I'm dealing with, so I prefer the Unicode code charts' capitalized ASCII descriptions to glyphs or empty boxes. Fortunately the unicodedata module has all this information available.

To get the output I wanted I came up with a little script:

# mockprint.py - wrapper around print
# function to handle
# UnicodeEncoding errors

# python 3.1

import unicodedata

ERRORSTR = "'ascii' codec can't encode character "
CHARIDX = 5
POSITIDX = 8
POSITIDX2 = 7

def mockprint(stringx):
"""
Wrapper for print() function that
replaces unprintable characters
with their Unicode names.
"""
try:
print(stringx)
except UnicodeEncodeError as e:
# main cases:
# 1) one character can't be printed
# 2) multiple characters in a row can't be printed
# 3) unicode character is first or last in string
# 4) other ascii characters surround the unicode ones
reasonx = str(e)
reasonx = reasonx.split(' ')
idx = reasonx[POSITIDX]
# more than 1 char in a row can't be printed
if idx == 'ordinal':
idx = int(reasonx[POSITIDX2][0])
if idx != 0:
print(stringx[:idx])
print(unicodedata.name(stringx[idx]))
mockprint(stringx[(idx + 1):])
# offending character shows up after ascii chars
elif len(stringx) > 1:
charx = int(reasonx[CHARIDX][3:-1], 16)
charx = chr(charx)
print(unicodedata.name(charx))
mockprint(stringx[(int(idx[0]) + 1):])
# end of the line
elif len(stringx) == 1:
charx = int(reasonx[CHARIDX][3:-1], 16)
charx = chr(charx)
print(unicodedata.name(charx))

A quick demo:

>>> import mockprint
>>> mockprint.mockprint('hello\u0401\u0402\u0403\u0404world')
hello
CYRILLIC CAPITAL LETTER IO
CYRILLIC CAPITAL LETTER DJE
CYRILLIC CAPITAL LETTER GJE
CYRILLIC CAPITAL LETTER UKRAINIAN IE
world

And something a bit more challenging:

A few foreign words in a number of different languages.

>>> fle = open('/home/carl/pythonblog/foreignbytestest', 'rt', encoding = 'UTF-8')
>>> import mockprint
>>> for linex in fle.readlines():
...     mockprint.mockprint(linex)
...
CJK UNIFIED IDEOGRAPH-65E5
CJK UNIFIED IDEOGRAPH-672C
CJK UNIFIED IDEOGRAPH-8A9E

abcde

ETHIOPIC SYLLABLE GLOTTAL A
ETHIOPIC SYLLABLE MAA
ETHIOPIC SYLLABLE RE
ETHIOPIC SYLLABLE NYAA

ARMENIAN CAPITAL LETTER HO
ARMENIAN SMALL LETTER AYB
ARMENIAN SMALL LETTER YI
ARMENIAN SMALL LETTER ECH
ARMENIAN SMALL LETTER REH
ARMENIAN SMALL LETTER ECH
ARMENIAN SMALL LETTER NOW

ORIYA LETTER O
ORIYA LETTER DDA
ORIYA SIGN NUKTA
ORIYA VOWEL SIGN I
ORIYA LETTER AA

LAO LETTER PHO TAM
LAO VOWEL SIGN AA
LAO LETTER SO SUNG
LAO VOWEL SIGN AA
LAO LETTER LO LOOT
LAO VOWEL SIGN AA
LAO LETTER WO

CYRILLIC SMALL LETTER ER
CYRILLIC SMALL LETTER U
CYRILLIC SMALL LETTER ES
CYRILLIC SMALL LETTER ES
CYRILLIC SMALL LETTER KA
CYRILLIC SMALL LETTER I
CYRILLIC SMALL LETTER SHORT I

CYRILLIC SMALL LETTER YA
CYRILLIC SMALL LETTER ZE
CYRILLIC SMALL LETTER YERU
CYRILLIC SMALL LETTER KA

Well, if that isn't beautiful, I don't know what is.

Seriously, this is a hack - parsing an error string and working backwards? I've got to be joking. Actually, no. For as much time as I've spent remembering after the fact that I can't print Unicode in the console, this is worth it, even if it's only good for Python 3.1.

py-openbsd's DoubleAssociation

2010-02-02T21:13:00.000-08:00

I briefly covered this structure last time, but didn't do it justice. The idea of a two-way dictionary structure (keys and values are both keys) intrigued me. I wanted to give it a spin with a real world example.

I've chosen a simple example with a few domain name (common names) and ip addresses:

# dblassoc.py

import openbsd

# some ip addrsses paired with domains
ips = {'google':(0x4a7d1393, 0xd8ef3d68),
       'openbsd':(0x8ef40c2a,),
       'freebsd':(0x45935321,),
       'yahoo':(0xd1bf5d34, 0xd183249e)}

# OK, we can now make the DoubleAssociation
ipsbothways = openbsd.utils.DoubleAssociation(ips)

print "ipsbothways['yahoo'] = " + str(ipsbothways['yahoo'])

# fair enough, but nothing we couldn't get from the dictionary

# try to query on an ip address to get a domain name
print "ipsbothways[(0x8ef40c2a,)] = " + ipsbothways[(0x8ef40c2a,)]

# unlike a normal dictionary, DoubleAssociation gives everything
# back with the keys() method

for keyx in ipsbothways.keys():
    try:
        if 0xd183249e in keyx:
            print "domain is " + ipsbothways[keyx]
            break
    except TypeError:
        print "TypeError: " + keyx

Python 2.5.4 (r254:67916, Jul 1 2009, 11:37:21)
[GCC 3.3.5 (propolice)] on openbsd4
Type "help", "copyright", "credits" or "license" for more information.
>>> import dblassoc
ipsbothways['yahoo'] = (3518979380L, 3515032734L)
ipsbothways[(0x8ef40c2a,)] = openbsd
TypeError: google
TypeError: openbsd
TypeError: yahoo
domain is yahoo
>>>

The one thing you have to look out for is the treatment of everything in the structure as a key - that's why I had to catch the TypeError. Everything is a value, too. The values and keys methods yield the same results.

In real life, if you had 30 or 50 or 1000 ip addresses, this would come in handy. Likewise for doctor-patient records, etc. (although the grouping of patients has to be unique, so that may not work after all - best to test both "sides" of the structure for exclusivity).

OpenBSD and Python

2010-01-31T17:55:00.000-08:00

Last time we covered FreeBSD's third party module, freebsd; this time we'll take a quick look at the equivalent openbsd package for the OpenBSD operating system.

$ python2.5
Python 2.5.4 (r254:67916, Jul 1 2009, 11:37:21)
[GCC 3.3.5 (propolice)] on openbsd4
Type "help", "copyright", "credits" or "license" for more information.
>>> import openbsd
>>> dir(openbsd)
['__builtins__', '__doc__', '__file__', '__name__', '__path__', '_ifconfig', '_netstat', '_packetDescriptors', '_pcap', '_sysvar', 'arc4random', 'ifconfig', 'netstat', 'packet', 'pcap', 'utils']

Let's see what all is hidden in that utils item:

>>> dir(openbsd.utils)
['DoubleAssociation', '__builtins__', '__doc__', '__file__', '__name__', 'cksum16', 'ethToBytes', 'ethToStr', 'findLongestSubsequence', 'getBlocks', 'ip6FromPrefix', 'ip6ToBytes', 'ip6ToStr', 'ipFromPrefix', 'ipToBytes', 'ipToStr', 'isIP6Addr', 'isIPAddr', 'isStringLike', 'multichar', 'multiord']

OK, a fair number of network addressing related functions.

help(openbsd.utils.ipFromPrefix)

ipFromPrefix(prefix)
    Produce an IPv4 address (netmask) from a prefix length.

That sounds handy. Let's give it a shot:

>>> openbsd.utils.ipFromPrefix(24)
'255.255.255.0'

>>> help(openbsd.utils.DoubleAssociation)

Help on class DoubleAssociation in module openbsd.utils:

class DoubleAssociation(__builtin__.dict)
| A double-association is a broadminded dictionary - it goes both ways.
|
| The rather simple implementation below requires the keys and values to
| be two disjoint sets. That is, if a given value is both a key and a
| value in a DoubleAssociation, you get unexpected behaviour.
|
| Method resolution order:
|      DoubleAssociation
|      __builtin__.dict
|      __builtin__.object
|
| Methods defined here:
|
| __init__(self, idict=None)
|      # FIXME:
|      #   While DoubleAssociation is adequate for our use, it is not entirely complete:
|      #       - Deletion should delete both associations
|      #       - Other dict methods that set values (eg. setdefault) will need to be over-ridden.

This one is kind of interesting - let's have a look:

>>> d = {1:'a', 2:'b', 3:'c'}
>>> d.get(1)
'a'
>>> print d.get('a')
None
>>> da = openbsd.utils.DoubleAssociation(d)
>>> da.get(1)
'a'
>>> da.get('a')
1

Just like the doc described it. Both the keys and the values are keys, if that makes sense.

Back up to the main modules of the openbsd package:

>>> help(openbsd.arc4random)

NAME
    openbsd.arc4random

FILE
    /usr/local/lib/python2.5/site-packages/openbsd/arc4random.so

FUNCTIONS
    getbytes(...)
        Get some random bytes.

And the result -

>>> bytesx = openbsd.arc4random.getbytes(10)
>>> [bytex for bytex in bytesx]
['\xb4', '\xd1', '\x86', '\xb7', 'g', '8', '\x10', '}', '\x8b', '\xe5']

One last module on a more common theme:

NAME
    openbsd.ifconfig - A Python module for querying and manipulating network interfaces.

FILE
    /usr/local/lib/python2.5/site-packages/openbsd/ifconfig.py

CLASSES
    __builtin__.int(__builtin__.object)
        FlagVal
    __builtin__.object
        Flags
        IFConfig
        Interface
        MTU
        Media
        Metric
    exceptions.Exception(exceptions.BaseException)
        _ifconfig.IfConfigError

    class FlagVal(__builtin__.int)
     | Method resolution order:
(etc.)


>>> intx = openbsd.ifconfig.Interface('rl0')
>>> print intx
rl0: flags=8843 mtu 1500
         media: Ethernet autoselect
         link: 00:30:bd:72:6a:a0
         inet6: fe80:2::230:bdff:fe72:6aa0
         inet: 192.168.100.100
>>> dir(intx)
['Iftype', 'Name', '__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', '_addrToStr', '_addrTypeLookup', '_getAddresses', '_getinfo', '_setflags', '_setmetric', '_setmtu', 'addAddress', 'addresses', 'delAddress', 'flags', 'media', 'metric', 'mtu', 'setAddress']
>>> intx.media
media: Ethernet autoselect
>>> intx.addresses
[{'address': {'sa_family': 18L, 'iftype': 'ETHER', 'address': '00:30:bd:72:6a:a0'}}, {'netmask': {'sa_family': 24L, 'address': 'ffff:ffff:ffff:ffff::'}, 'address': {'sa_family': 24L, 'address': 'fe80:2::230:bdff:fe72:6aa0'}}, {'netmask': {'sa_family': 0L, 'address': None}, 'dstaddr': {'sa_family': 2L, 'address': '192.168.100.255'}, 'address': {'sa_family': 2L, 'address': '192.168.100.100'}}]
>>>

ifconfig available within Python - sweet. rl0 is the ethernet device on my old Dell tower.

Examination of the openbsd package shows that it has quite a bit to offer. If you're using OpenBSD, there's nothing stopping you from doing routine sysadmin tasks with Python. If not, now you've got a reason to check it out.

Python Modules for the BSD's

2010-01-31T14:42:00.000-08:00

Well, for FreeBSD and OpenBSD, at least. I can't yet vouch for NetBSD and Dragonfly BSD.

First, FreeBSD - the port is named py-freebsd. Once built, the module can be imported with "import freebsd".

[carl@pcbsd]/usr/local/lib/python2.6/site-packages(158)% python
Python 2.6.2 (r262:71600, Jun 24 2009, 23:31:28)
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import freebsd
>>> dir(freebsd)
['__doc__', '__file__', '__name__', '__package__', '__version__', 'chflags', 'const', 'fchflags', 'fstatfs', 'geom_getxml', 'getfsent', 'getfsfile', 'getfsspec', 'getfsstat', 'gethostname', 'getloadavg', 'getlogin', 'getosreldate', 'getpriority', 'getprogname', 'getpwent', 'getpwnam', 'getpwuid', 'getquota', 'getrlimit', 'getrusage', 'ifstats', 'ipstats', 'jail', 'kevent', 'kqueue', 'ktrace', 'lchflags', 'quotaoff', 'quotaon', 'quotasync', 'reboot', 'sendfile', 'sethostname', 'setlogin', 'setpriority', 'setproctitle', 'setprogname', 'setquota', 'setrlimit', 'statfs', 'sysctl', 'sysctldescr', 'sysctlmibtoname', 'sysctlnametomib', 'tcpstats', 'udpstats']

Not a bad collection of utilities. Let's take a couple for a test drive:

>>> freebsd.gethostname()
'pcbsd'

>>> freebsd.getprogname()
'python'
>>> help(freebsd.jail)
Help on built-in function jail in module freebsd:

jail(...)
jail(path, hostname, ip_number):
The jail() system call sets up a jail and locks the current process
in it. The ``path'' should be set to the directory which is to be
the root of the prison. The ``hostname'' can be set to the hostname
of the prison. This can be changed from the inside of the prison.
The ``ip_number'' can be set to the IP number assigned to the prison.

>>> # wow, you can set up a jail with python

>>> freebsd.ifstats()
>>> >>> import pprint
>>> pprint.pprint(_)
{'bge0': {'addrlen': 6,
'baudrate': 100000000L,
'collisions': 0L,
'flags': 34883,
'hdrlen': 14,
'hwassist': 7L,
'ibytes': 19222590L,
'ierrors': 0L,
'imcasts': 577L,
'ipackets': 19728L,
'iqdrops': 0L,
'metric': 0L,
'mtu': 1500L,
'name': 'bge0',
'noproto': 0L,
'obytes': 2009038L,
'oerrors': 0L,
'omcasts': 0L,
'opackets': 13285L,
'pcount': 0,
'physical': 0,
'snd_drops': 0,
'snd_len': 0,
'snd_maxlen': 511,
'type': 6},

bge0 is the ethernet device on my Thinkpad.

>>> freebsd.getlogin()
'carl'
>>> freebsd.tcpstats()
>>> pprint.pprint(_)
{'accepts': 0L,
'badsyn': 0L,
'cachedrtt': 147L,
'cachedrttvar': 150L,
'cachedssthresh': 4L,
'closed': 495L,
'connattempt': 360L,
'conndrops': 20L,
'connects': 340L,
'delack': 277L,
'drops': 22L,
'keepdrops': 0L,
'keepprobe': 0L,
'keeptimeo': 0L,
'listendrop': 0L,
'mturesent': 0L,
'pawsdrop': 0L,
'persistdrop': 0L,
'persisttimeo': 0L,
'predack': 0L,
'preddat': 15226L,
'rcvackbyte': 1093284L,
'rcvackpack': 1848L,
'rcvacktoomuch': 0L,
'rcvafterclose': 7L,
'rcvbadoff': 0L,
'rcvbadsum': 0L,
'rcvbyte': 16595286L,
'rcvbyteafterwin': 0L,
'rcvdupack': 232L,
'rcvdupbyte': 88723L,
'rcvduppack': 77L,
'rcvoobyte': 1015050L,
'rcvoopack': 919L,
'rcvpack': 15882L,
'rcvpackafterwin': 0L,
'rcvpartdupbyte': 525L,
'rcvpartduppack': 2L,
'rcvshort': 0L,
'rcvtotal': 18489L,
'rcvwinprobe': 0L,
'rcvwinupd': 3L,
'rexmttimeo': 118L,
'rttupdated': 1817L,
'sc_aborted': 0L,
'sc_added': 0L,
'sc_badack': 0L,
'sc_bucketoverflow': 0L,
'sc_cacheoverflow': 0L,
'sc_completed': 0L,
'sc_dropped': 0L,
'sc_dupsyn': 0L,
'sc_recvcookie': 0L,
'sc_reset': 0L,
'sc_retransmitted': 0L,
'sc_sendcookie': 0L,
'sc_stale': 0L,
'sc_unreach': 0L,
'sc_zonefail': 0L,
'segstimed': 1688L,
'sndacks': 9261L,
'sndbyte': 1098259L,
'sndctrl': 697L,
'sndpack': 1252L,
'sndprobe': 0L,
'sndrexmitbyte': 2252L,
'sndrexmitpack': 2L,
'sndtotal': 12381L,
'sndurg': 0L,
'sndwinup': 1169L,
'timeoutdrop': 9L}

22 drops, 9 of them timeouts, and a bunch of other stuff too.

Enough for today. Next time we'll take a quick look at the Python module for OpenBSD.

Python Logos III: Pycon India 2009 contest

2010-01-28T05:39:00.000-08:00

Just received an e-mail from Noufal this morning (actually it was a comment on one of the previous two logo posts) about the Indian Pycon logo contest last year. I've already shown the winner in a previous post, but the entries are worth seeing in their own right. Hats off to the contributors:

** SPECIAL NOTE: Some of the following logos do not comply with the Python Software Foundation's trademark rules for the use of the logo. Although there was no harm intended through use in such manner, it's always good to keep the full logo intact (don't split the snakes or take one away!). My very basic, and perhaps inaccurate understanding is that it's OK to use the logo and even change it's colors, but the shape should remain intact.

Python Logos Part II: Free Form

2010-01-23T04:51:00.000-08:00

Free Form Python Logos

The Many Faces of the Python Logo

2010-01-22T18:56:00.000-08:00

Variations on Standard Python Logo

Free form logos next time . . .

Python 3.1 String Formatting - Positional Arguments

2010-01-11T15:36:00.000-08:00

Last time I did a goofy little trick with the new Python 3 string formatting. This time I'd like to demo something a little more practical.

I had heard a lot about positional arguments being more flexible and easier in Python 3's string formatting. The basic gist is that the positions being referred to are on the right side of the string's format method as opposed to the left. This usually results in fewer positions, and, hopefully, less confusion.

Quickie example:

>>> class Foo:
>>>    def __init__(self):
>>>        pass
>>>
>>> a = Foo()
>>> a.foo = 24
>>> a.bar = 33
>>> a.baz = 6
>>> # Python 2 string formatting
>>> # print '%d %d %d' % (a.foo, a.bar, a.baz)
>>> print('{0.foo} {0.bar} {0.baz}'.format(a))

24 33 6

It's true that the left side is a bit more involved now with the new string formatting. There are more options for formatting, which I have omitted. My main goal here is to understand the new formatting to the point where I can leverage it to my advantage in my work. Let's see what, if anything, I've learned.

In the mining industry I did a lot data tracking on various production metrics - flows through pipes, broken rock moved to its final location, explosive power, etc. One of the most challenging things to track was production in the open pit. It all centered around big multimillion dollar electric shovels.

Anyway, to make a long story longer, by the time you worked your way up the object hierarchy to the shovel, you ended up with about 20 or more "fields" in the database, or members of your shovel object. Smithing csv output from these objects could get pretty hairy, because there were so many fields to track, and they changed frequently.

Out of reality - it's just too complex - back to the programming world. This Python 3 string formatting could have helped out some in this situation. Let's go back to the Foo object:

UPDATE!
I had made an error in the print statment below by mixing quotes. It is now corrected.
Yet another example of why code must be tested :-\

>>> # make two more Foo objects
>>> b = Foo()
>>> b.foo = 'hello'
>>> b.bar = 'cruel'
>>> b.baz = 'world'
>>> c = Foo()

>>> c.foo = '99.234'
>>> c.bar = '23.715'
>>> c.baz = '18.6'
>>> for objx in a, b, c:
>>>     for strx in 'foo', 'bar', 'baz':
>>>         print(('{0} = {1.' + strx +
                            '}').format(
                           strx, objx))

foo = 24
bar = 33
baz = 6
foo = hello
bar = cruel
baz = world
foo = 99.234
bar = 23.715
baz = 18.6

Simple example, but it illustrates some of the utility of the new string formatting. If I were trying to track a bunch of data, I would only have to redefine my "fields" once in a global constant ['foo', 'bar', 'baz']. objx could be cycling (actually, iterating) through a list of objects. Any writing or crunching could be done in a separate (small) function or method outside the loop (since ['foo', 'bar', 'baz'] is constant).

Turns out this new string formatting is something I could get used to . . .   ;-)

Fun With Python 3.1's New String Formatting . . .

2010-01-07T18:58:00.000-08:00

Or a totally unoriginal poor ASCII art rendition of the Transamerica building at night . . .

carl@L733rLinux:~$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> # saw this on Python-Dev (Stephen Bethard?)
>>> format('HEADER', '*^50')
'**********************HEADER**********************'
# just for fun . . .
>>> for x in range(30):
...     print('{0:*^50}'.format(x * ' '))
...
**************************************************
************************ *************************
************************ ************************
***********************   ************************
***********************    ***********************
**********************     ***********************
**********************      **********************
*********************       **********************
*********************        *********************
********************         *********************
********************          ********************
*******************           ********************
*******************            *******************
******************             *******************
******************              ******************
*****************               ******************
*****************                *****************
****************                 *****************
****************                  ****************
***************                   ****************
***************                    ***************
**************                     ***************
**************                      **************
*************                       **************
*************                        *************
************                         *************
************                          ************
***********                           ************
***********                            ***********
**********                             ***********
>>>

If that isn't beautiful, I don't know what is . . .