Try repeating the title of this blog posting three times fast.
And if you think that is tough, try coding one. I haven’t been paid to write computer programs since I left the research faculty of the Albert Einstein College of Medicine back in 1986, so my coding skills could be a little rusty, although I have written lots of simple scripts, modified dozens of others in multiple languages and help programmers debug their programs in more languages than I can count, but more about that in another post. This one is a rant about Python.
I was first introduced to Python as a programming language in 2001, when my boss at Bear Stearns at the time (who shall remain nameless) promised the London research team that he would speed up the delivery of their research Emails. At the time we were using Sendmail with some intervening shell scripts to manage our outbound Emails and the London research team was sending their research out to lists of several hundred addresses. Since Sendmail in those days was single-threaded the first addresses in the list got the research fairly quickly (by the standards of those days). Unfortunately the owners of the lists didn’t always keep them clean and up-to-date, so there were often bad addresses or addresses with misspelled domains. Since Sendmail would try every MX server for a domain, or the A record for the domain if there was no MX records, then every bad address or domain would slow down the delivery to subsequent recipient addresses. By the time Sendmail got to the end of the list the timely research was often obsolete so our London research team, and their customers, were understandably upset. My boss’ solution was to obtain 2 Unix servers, powerful Solaris boxes at the time, download Postfix, Python and Mailman onto those servers, and then hand them over to me and resign from the firm. It became my job to put all this together so that the London research team, and ultimately several other Bear Stearns teams, could use these servers to send out their research in a timely manner.
Mailman, for those of you who don’t know it, is open-source mailing list manager software written in Python. The then current version of Mailman did not include “Real name” support for members which I see is now a feature of the current version, but our users required it, since they couldn’t be bothered knowing the actual Email addresses of their clients. That version also didn’t include the concept of a list member manager separate from the list manager, although we wanted our research people to be able to maintain their mailing lists without having any access to the other features of their mailing lists. Thus, I had to write an entire new user interface for the Mailman mailing lists which allowed the list owners to import/add/delete real names and Email addresses for their clients but which hid from them the other features of Mailman and their lists. Fortunately Python is an object-oriented language and the Mailman lists were nested objects so it was not too difficult to add attributes to the list objects for real names, and to modify the user interface to restrict what our list owners could do. Of course, first I had to teach myself enough Python to understand the Mailman source code and figure out how to modify it. That took a couple of weeks. As I recall the Mailman code was written in Python 1.5, so things have changed a lot since then, but that was my introduction to Python.
Fast forward to 2019, where I’m helping an old friend from BS with some software she is writing in Python and she determines that she needs to be able to do Single Sign On (SSO) using SAML for one of her customers. This being the era of Linux, open-source software and shared library modules I searched for a Python module that could be used as a SAML Service Provider. I found a few, but none had adequate documentation to just plug them in and most were designed for specific web frameworks. My friend was writing her code in Python 2, using a web framework written by another old BS alumnus which mostly outputs JSON and was unable to supply the 302 status which the browser needed for simple HTTP redirects to the SAML IdP. Also, this being 2019 and the last year that Python 2 will be supported (although I see that there are still some utilities which may not be Python 3 ready), my code had to work with Python 2, but be upward compatible to Python 3. I managed to get a working proof of concept (POC) for the code using Apache and Python 2 CGI, but it is still clunky.
Moving the code from Python 2 to Python 3 has been more of a headache than anticipated, mostly because of the change in the way strings are handled. Distinguishing between byte strings and Unicode strings is very necessary, but it becomes a pain to manage when modifying lots of legacy code. But that’s not my major complaint about Python. Maybe my complaint is just because I haven’t taken the time to understand the issues involved, but it seems that the method Python uses for locating system modules has evolved over the last 20 years in not always compatible ways. The latest idea, of every application having its own environment with its own set of library modules may make sense in these days of really cheap memory and storage, but is difficult for us old-timers who are used to having limited memory to work with. Here again I will save this for another post.
I’ve had several utilities which are coded in Python and which self-update, but which have been unable to find their modules since the default Python was modified from Python 2 to Python 3, even if they have their own version of Python in their environments. I’m not sure how to tell these utilities how to find the commonly installed modules, or how to install needed modules into their specific environments. I’m sure I will figure this out in the next day or two, but it would have been great if Python was able to do it by itself without forcing me to go through these contortions to make things work.
Enough minor ranting for now, but I did make some promises above for more posts in the future. I’m hoping to put together more, shorter posts. We’ll see if I can do that.
Thanks for reading this.
So, I’ve done some Googling and some looking at library scripts and it seems that the problem is that each Linux distro has its own idea of where Python libraries should be stored, so you have /usr/lib/python$VER and /usr/lib/python$VER/site-packages and /usr/lib/python$VER/dist-packages and repeat each of those for /usr/lib64, and /usr/local/lib and /usr/local/lib64. Then throw in the individual user specific $HOME/.local/lib/python$VER/ and the application specific venv/lib/python$VER and venv/lib64/python$VER and I’m not surprised that it becomes difficult to find a specific library module that you want. Couple that with the self-updating utilities like pip, yum, certbot, etc. which sometimes change where they expect to find their modules and you have a fun situation.
But enough of my ranting for today.