PyGuile – version 0.1.0 has been released for beta testing!

UPDATE: At  22 May 2016, I uploaded the project to Github. You may find it in https://github.com/tddpirate/pyguile.

DISCLAIMER: the software has not been really finished and it is lacking some useful features such as Python exception handling by the Scheme side.

PyGuile – Part 4 – Argument and result conversion issues

There is no 1:1 mapping between Scheme and Python data types. As a consequence, there are several cases, in which PyGuile has to guess how would the user like to have the arguments and result of a Python function converted. Instead of guessing, we would like to empower the user to be explicit about the kinds of conversion which he wants.

The following is a census of ambiguous data conversion cases, which I identified.

  1. Scheme pair ->
    • Python 2-Tuple
    • Python 2-List
  2. Scheme list ->
    • Python Tuple
    • Python List
    • Nested tree of pairs (2-Tuples or 2-Lists)
  3. Python 2-Tuple or 2-List ->
    • Scheme pair
    • Scheme list
    • Scheme rational (if the Pythonic data structure consists of two integer values)
  4. Scheme alist (association list) ->
    • Python Dict
    • Python Tuple/List of 2-Tuples
  5. Python string ->
    • Scheme string
    • Scheme symbol
    • Scheme keyword

    Additional considrations:

    • Case sensitivity of symbols and keywords
    • String representation of keywords in Guile has leading dash – to retain or remove it in the Python side of affairs?
  6. Python 1-character string ->
    • Scheme char
    • Scheme string

    Additional considration: utf-8 encoded glyph is a sequence of few characters.

  7. Python int ->
    • Scheme int
    • Scheme bignum
    • Scheme char
  8. Python None -> One of several possible values: ‘(), #f, SCM_EOL, ‘*None* or another custom Scheme value.
  9. Python (),[],{} ->
    • Scheme ‘()
    • SCM_EOL
    • Custom Scheme value
  10. Scheme ‘() ->
    • Python ()
    • Python []
    • Python {}
  11. SCM_EOL ->
    • Python (),[],{}
    • Python None
    • Custom Python value
  12. Scheme rational ->
    • Python Float
    • 2-Tuple of Python Ints
  13. Scheme exact/inexact flag in numerical values – if and how to represent it in the Python side of the application?
  14. Giant data structures with sparse access needs – lazy vs. eager conversion
  15. Exception objects
  16. Objects of certain classes (vectors, ports, functions, images, etc.)

There is also the separate issue of string encoding/decoding, with which we deal by mandating that anything passing between Scheme and Python has to be utf-8 encoded.

One of the goals of PyGuile is to make it efficient to invoke Python library functions from Guile. Therefore, efficiency of conversion of function arguments and results is critical.

When there are no user hints, the following inefficiencies occur:

  1. PyGuile has to make a default (and possibly sub-optimal) choice when encountering one of the above ambiguous cases. Then the script using the data has to reformat it to match the data format to its actual needs.
  2. PyGuile has to identify the data type of each datum. The present implementation does not go into the internal representation of Guile (SCM) and Python (PyObject) objects, therefore PyGuile has to test for various data types one by one, until one of them matches the argument.
  3. Sometimes a Python procedure needs to do no processing on one of its arguments. The argument’s value needs only to be passed around as a pointer, or to be inserted into the right place in a result data structure. In such a case, it is desirable to use the most efficient conversion possible i.e. wrap/unwrap opaque objects. This is a generalization of the case of giant data structures with sparse access needs.

Therefore, when performance is critical, hints from the user would help not only to disembiguate the conversion process but also to speed it up.

The user hints will be implemented as follows.
With each function (Python function invoked from Guile, or Guile function invoked from Python) we associate two (possibly degenerate) signatures. One signature will contain the hints for converting the function’s arguments. The second signature will hint how to convert the function’s result. The signatures are Scheme lists, whose leaf nodes are symbols denoting conversion functions.

Chris Jester-Young, in his answer to my question in Stackoverflow, proposed the following function for traversing two corresponding tree structures, and applying the functions in one of them to data in the other one.

  (define (map-traversing func data)

    (if (list? func)

        (map map-traversing func data)

        (func data)))

Using it requires unquoting. Example:

  (map-traversing `((,car ,cdr) ,cadr) '(((aa . ab) (bb . bc)) (cc cd . ce)))

Our implementation will differ from the above in details, as the signatures’ leaf nodes do not denote proper Scheme functions.

PyGuile – Part 3 – Non-goals

Every project, including software development projects, needs an identity. It needs a definition of its boundaries. It has to be clear about what is inside it and what is outside of it.

Without such a definition, the project would try to be too many things for many people, and as a result, its products would not be really useful for anyone.

A project’s identity makes it easier and faster to make design and trade-off decisions.

Given the above trite introduction, and given that there is a list of goals for the PyGuile project, a list of non-goals is needed as well and here it is.

  1. Theoretical academic purity – attempt to convert every data type from Guile to Python and vice versa, and to support the whole range of values assumed by each data type.
  2. Ability to mix code snippets from both Scheme and Python in the same source code file.
  3. Invocation of machine language libraries (static or DLLs) – for this purpose, there are already existing tools (SWIG and PerlXS).
  4. Framework for making it easy to add support for interoperation with yet another scripting language.

There are also some goals, which are low priority and I do not plan to shed tears if they prove to be impossible to achieve without significant effort:

  1. Thread-safety
  2. Tail recursion support

PyGuile – Part 2 – Design Issues

While working on the PyGuile, I identified the following design issues.

  1. The data type trees of Scheme and Python do not have an 1:1 correspondence.
    • Do we want to convert a Scheme list into a Python Tuple or a Python List?
    • How about an alist (associative list) – should be a Python List of 2-tuples or a Python Dict?
    • And in the other direction – do we want to convert a Python string into a Scheme string, symbol or keyword?
  2. API for adding plugins which convert between Guile and Python representations of useful data types (such as file handles, images or Berkeley sockets).
  3. How do we want to pass large data structures – convert them immediately, or employ lazy conversion (convert an element only when it is requested)? If we employ lazy conversion, how do we implement the associated bookkeeping? See more about this below.
  4. How do we deal with the different garbage collection regimes of Guile and Python? In particular, how do we make SCM objects owned by Python objects known to the Guile garbage collector?
  5. How will we support Unicode? Bear in mind that we want to minimize manipulations of long text strings.
  6. How to allow each scripting language to seamlessly invoke functions in the other scripting language?

The problem of lack of 1:1 correspondence will be dealt with as follows.

A standard conversion convention, which will work for the overwhelming majority of cases, will be employed. Functions, which have special needs, will have their argument conversions specified by means of a suitable tree-structured template.

When passing a data structure (or object) created in language A to language B, the following cases can happen:

  1. Opaque pointer – B only passes it around. A performs all processing and B just holds the pointer for future reference.
  2. B accesses a single element (or small number of elements) in the data structure.
  3. B loops over all elements of the data structure.
  4. B needs arbitrary access to several elements of the data structure (example: image processing).

Those cases can be dealt with as follows:

  • Case 1 can be handled by wrapping a language A pointer by a language B object, which carries opaque data around.
  • Cases 2,3 can be dealt by means of custom data access procedures (such as Python’s __getitem__()). An element will be converted only when it is actually requested. Elements in nested data structures can be dealt with as in case 1.
  • Case 4 can be handled by implementing a mechanism for plugging in and registering custom conversion functions for specific data types.

In practice, the most tough design issue, which I identified so far, is the management of the SCM objects owned by Python objects.

When a SCM object is assigned to an attribute of a Python object, some registration mechanism needs to
be invoked so that the SCM object can be reclaimed by the Guile garbage collector if the Python object goes out of scope. The registration mechanism needs also to take care of marking the SCM objects while they are owned by a living Python object.

PyGuile – Part 1 – Using Python libraries in Guile (a Scheme implementation) scripts

For long time I have dreamt of invoking Python libraries from scripts written in Scheme. The reason for this is to be able to enjoy the fantastically rich control structures possible in Scheme, yet use familiar libraries to accomplish useful actions, some of which are unavailable in SLIB and other Scheme libraries.

Now at last I am working on realizing this dream. The Scheme implementation being used is version 1.6 of Guile and the Guile extension being developed embeds a Python 2.4 interpreter. In the future, more recent versions of Guile and Python will be used.

The goals of the project are:

  1. Make it easy to invoke Python libraries from Guile.
  2. The integration between Python and Guile is to be seamless.
  3. The architecture of the implementation shall enable optimizations for efficient runtime behavior.

To accomplish those goals, it is necessary to:

  1. Convert primitive Scheme data types (integers, reals, Booleans, strings, lists) into the corresponding Python data types, and vice versa.
  2. Be able to invoke functions defined in one language from the other language. This has to be bidirectional in order to support callbacks.
  3. Be able to pass around pointers to objects (as opaque values) and invoke methods over them.
  4. Have efficient transfer of control and data between both languages.
  5. Deal with different garbage collection conventions in both environments.
  6. Be able to optimize code for a particular pair of language runtime systems.
  7. Nice to have: support for recursion, especially tail recursion.
  8. Nice to have: thread-safety.

It is envisioned that the software developed in this project will be part of a larger system, which will allow more scripting languages to interoperate with Guile and with each other.

There is another project – Schemepy – which embeds a Scheme interpreter in Python scripts.  This project has different focus and it essentially allows Scheme to be used for those parts of a project, in which its strengths are especially important.

A Vista Conspiracy Theory

One possible reason for the stupidity of Microsoft in handling MS-Vista, especially in its attempts to ram MS-Vista through its customers’ throats instead of MS-Windows XP, is as follows.

Shortly after SCO sued IBM and other companies due to violation of its Linux copyright, IBM and possibly other big companies decided upon two-pronged counter attack.  First, they would fight SCO in court to the bitter end.

The conspiracy theory expoused below has to do with the second prong.  The goal here is to cause Microsoft to bleed as much money and as quickly as possible, so that it’ll not have the financial means to continue to support SCO until its defendants wear out.

For this purpose, moles may have been installed in Microsoft (or maybe Microsoft employees were bribed) to deliberately make the wrong managerial decisions, to sap the morale of the working software developers, to entangle the projects in cobwebs, to bog the projects down in intricate dependencies and frivolous compatibilities with the past, to surrender too easily to Hollywood moguls when they ask for DRM measures to be built into MS-Vista.

Since Microsoft had the fatal combination of de-facto monopoly position and huge cash reserves, both had to be attacked.  The monopoly position was attacked by making MS-Vista incompatible with MS-Windows XP, so that people would find it just as easy to switch to Linux or to Mac OS as it is to MS-Vista.  The cash position was attacked by turning MS-Vista into huge cash drain.

A new software developers’ mutual help Web site (no longer) rudely excludes deaf software developers

The newly announced http://www.stackoverflow.com/ Web site confines all communications to the audio format. No provision for textual transcription of the audio podcasts exists. Users’ submissions are accepted only if they are in audio format. This is probably the founders’ newest idea for filtering out spam and flames.
However, it is a case of rude inaccessibility. Please do not contribute and do not browse the Web site – and let the founders know your opinion about this case of throwing out the baby with the bathwater.
The announcements in the founders’ blogs are as follows:

What next, a Web site, which excludes gay software developers?

23 APR 2008 UPDATE:

The podcasts are now transcribed into text, making them accessible to the deaf as well as being helpful to people, who want to discover them using search engines, and people having no time to listen through the entire podcast.
The transcription mechanism is Wiki-based, allowing people to transcribe text piece by piece. So even if you have only 15 minutes to spare, you can still make a contribution.
It is still necessary to persuade them to accept questions as text in addition to sound clips…

Choosing a Python module for accessing Microsoft SQL Server Unicode data

One day I found myself in need of Python code, which retrieves Unicode data from Microsoft SQL Server tables. The code needs to run on a PC with MS-Windows XP.

The dbi and odbc modules, which I used in the past, failed miserably in this task, by forcing the Unicode data to be converted into string data, using the ascii encoder.

So, I had to look for other Python modules. My findings from evaluating the relevant Python modules are summarized below.

dbi,odbc from pywin32
  • Package: pywin32-210.win32-py2.5.exe, available from Python for Windows Extensions.
  • Textual data is passed as strings, rather than as Unicode.
  • Parameters in SQL queries are marked by ‘?’.
  • Dates/times are retrieved as instances of the dbi.dbiDate class (essentially, a wrapped long int).
win32com
I was not successful in using the win32com based code, which worked for
Arik Baratz. According to him, this code uses the Microsoft ActiveX Data Objects 2.8 Library. It requires the modified version 209.1 of pywin32, which comes with version 2.5.1.1 of the ActiveState Python distribution. This modified version adds to the win32com class an extra member – client.
You need to add the following line sometime after the import win32com:

win32com.client.gencache.EnsureModule('{DA9159C1-C9D5-4223-81AC-2B1080701D12}',0,1,0)

To actually start working, use win32com.client.Dispatch() to establish a connection to the SQL Server.

pymssql
pyodbc
  • Package: pyodbc-2.0.39.win32-py2.5.exe, available from pyodbc – A Python DB API module for ODBC
  • Textual data is passed as Unicode.
  • Parameters in SQL queries are marked by ‘?’.
  • Dates/times are retrieved as instances of the datetime.datetime class.

The Python module chosen is pyodbc.

Are you the Webmaster of an IE-only Web site?

Then the following is a must read for you:

How Not To Do Market Research

Web sites, which support only IE, would not be visited by people, who use other browsers. So OF COURSE, they would not “report enough traffic to justify” support for W3C standards and/or other browsers.

By the way, my Web site’s browser statistics since the start of November 2007 indicate the following browser percentages (disclaimer for Israeli Webmasters: my Web site’s audience is international):

MS Internet Explorer 42.6%
Mozilla 31.7%
Unknown 12.5%
Firefox 9.7%
All the rest 3.5%

Technical Debt

When developing software, there are several times at which you can choose among quick&ugly hack, which will incur higher maintenance headaches in the future, and slow&clean design. Usually, the slow&clean design is the way to go. But life is complicated and has a way to contrive an exception for each rule. Therefore, one should know how to manage the consequences of quick&ugly hacks.

This is the subject of technical debt, written about by people like Steve McConnell.

Roughly, you incur technical debt whenever you make a design or implementation decision which will require future rework or higher maintenance workload. The decision could be also something trivial such as not bothering to invest in documenting your present design, causing the future maintainer to waste time learning your design before modifying it.

Technical debt is written off when the software package in question is taken out of use due to replacement by a newer software package, the application area becoming irrelevant, or the company going bankrupt.

References: