Category: Python

Include fragments in static site source files to be processed by Pelican

I use Pelican to generate static websites using markdown files.

I found that there are text fragments that I want to reuse in several of my web pages, but not in all of them. So I needed a way to include those fragments in the markdown pages, using a suitable include directive, rather than add them to the template files used by Pelican to generate the final HTML files.

I googled for a way to use include files in markdown files to be processed by Pelican. I was led to unhelpful answers. For example, the mdx_include module. I rejected it because I found no easy way for lazy people to integrate it into Pelican.

Eventually I converged upon the following solution.

Pelican uses the Jinja2 templating engine, which has the {% include 'filename' %} directive.

By default, Pelican subjects only template files to Jinja2 processing. So, to use the include directive in my own markdown files, I need a way to have them go through Jinja2 processing as well.  This is what the jinja2content Pelican plugin does.

To install it:

pip install pelican-jinja2content

To use it, add the following to your `pelicanconf.py` file:

PLUGINS = [
.
.
.
'pelican.plugins.jinja2content',
.
.
.
]

# jinja2content configuration

IGNORE_FILES = [
'includes'
]
JINJA2CONTENT_TEMPLATES = [
'./includes',
]

Then add your include files to the ./content/includes subdirectory. They may have any extension you want including .md. I recommend .mdinc to tell them apart from your article markdown files. Instead of includes, you may use any subdirectory name by changing the configuration above.

Before doing sophisticated things with your include files, read the plugin’s documentation and understand the limitations of using Jinja2 tags inside your include and markdown files.

Fun with Python’s “import this”

Everyone knows that when you perform import this, in your Python script, you get a printout of The Zen of Python, by Tim Peters.

However, what else is there in the this module?

>>> import this
The Zen of Python, by Tim Peters
.
.
.
>>> help(this)
Help on module this:

NAME
    this

MODULE REFERENCE
    https://docs.python.org/3.8/library/this
.
.
.
DATA
    c = 97
    d = {'A': 'N', 'B': 'O', 'C': 'P', 'D': 'Q', 'E': 'R', ...
    i = 25
    s = "Gur Mra bs Clguba, ol Gvz Crgref\n\nOrnhgvshy vf o...
FILE
    ...

It looks like The Zen of Python text is stored in the module as this.s in ROT-13 encrypted form. The table for decoding this text is stored in this.d.

What is the most Pythonic way to decode the text in this.s?

  1. Loop over the characters in this.s: 
    [something(c) for c in this.s]
  2. Decode c if it is encoded (is in the this.d dictionary): 
    [(this.d[c] if c in this.d else c) for c in this.s]
  3. The result of the comprehension is a list of characters. Make a string out of it: 
    ''.join([(this.d[c] if c in this.d else c) for c in this.s])
  4. Now, print the result: 
    print(''.join([(this.d[c] if c in this.d else c) for c in this.s]))

That’s all, folks!

Automatic localtime management in ESP8266 and other low-memory IoT devices

Justification

During the last several years, personal computers and smartphones became capable of displaying the local time, correctly adjusted for daylight saving time (DST) – and without requiring human intervention beyond selecting the correct timezone.

Nowadays, there are also some IoT devices, which need to support local time management – displaying it, or otherwise making it available.

Timekeeping is performed using the Internet protocol NTP, which provides the correct UTC. When using a PC or a smartphone, the timezone is usually selected by manual user action.

However, some IoT devices may not have the UI needed for convenient timezone selection. Then it is desirable to support automatic timezone selection as a default.

How to implement automatic timezone selection?

There are some websites, which discover your IP address and provide you with the best guess of your timezone.

Since those websites usually provide the timezone name rather than the string describing the DST transition dates (the so-called tz_string), the next step is to figure out the DST transition dates from the timezone name.

In devices with plenty of memory this is carried out by means of a timezone database.

For example, in Debian/Ubuntu based systems, this database is stored in the /usr/share/zoneinfo directory and occupies 3.5MB (the relevant package in Ubuntu 18.04 is tzdata and its version, as of Nov. 2019, is 2019c-0ubuntu0.18.04).

Memory constrained IoT devices

However, IoT devices are typically based upon memory-constrained controllers and cannot afford to store locally the whole timezone database – just to correctly determine the local time for a single timezone.

Therefore, IoT devices need to access an Internet based service to get the correct timezone information, just as they get UTC time updates using NTP. In other words, those IoT devices effectively outsource the timezone database management.

Internet service for providing the timezone information

An Internet service, for providing the correct tz_string corresponding to a timezone name, needs to keep the timezone database up to date at all times.

I implemented the internet service as follows.

  1. A machine, running an Ubuntu 18.04 installation with a webserver, is used.
  2. The Internet service is implmented as a small WSGI-based website. It uses the database mentioned below.
  3. A script scans the /usr/share/zoneinfo contents and creates a small database for translating timezone names into the corresponding tz_string values.
  4. There is a mechanism for invoking the above script and restarting the web server each time the tzdata package is updated/installed/re-installed.

Show us the code!

The GitHub project tddpirate/tzdata2tzstring includes redacted versions of both an implementation of the above website and a sample ESP8266 client.

Credits

I wish to thank the Python Israel Telegram group members for advice about selecting a Python WSGI framework. I ended up selecting Falcon because benchmarks indicated that it is faster than Flask and Bottle.

The members of the לינוקס Telegram group deserve thanks, too. They helped me find the mechanism for appending my own postprocessing scripts after a Debian/Ubuntu package installation or upgrade.

Python discovers its inner PHP and JavaScript personae

Did you recently switch from PHP or JavaScript to Python, and are missing the fun of being bitten by your programming language?

The collection of surprising Python snippets and lesser-known features is your ultimate guide for provoking Python to bite you in the arse.

I got the ability to work with Heroku using my Debian Stretch system

The other day I found that:

  1. Heroku needs Python 3.6 or later to work (as of June 22, 2018). See: Getting Started on Heroku with Python.
  2. Debian Stretch (Debian Stable as of June 22, 2018) and its backports have only Python 3.5.

The solution was to build a Docker image based upon Ubuntu 18.04, which does have Python 3.6. See the project https://gitlab.com/TDDPirate/heroku_on_debian in GitLab.

July 15, 2018 update:

After I complained about flakiness of Selenium-based tests when the Selenium server is running outside of the Docker container while the application runs inside the container, Udi Oron suggested another way to run Python 3.6 on a Debian Stretch system: use pyenv.

Turns out that pyenv solves the pain point of running Python 3.6 on Debian Stretch without having to use a container. So Selenium-based tests are now stable.

The following is an excellent article about using pyenv:
Pyenv – Python Version Management Made Easier

And the following is a link to the GitHub repository:
https://github.com/pyenv/pyenv

I suspect that pyenv is the reason why people are not in a hurry to backport new Python versions to Debian.

How to visually compare two PDF files? (cont’d)

When I asked the above question in a Telegram group, people proposed also other tools, which I am summarizing below.
Amiad Bareli, Amit Aronovitch, Meir Gil and Yehuda Deutsch – thanks.

  1. ImageMagick compare
  2. matplotlib testing framework – supports also PDF: 
    >>> import matplotlib.testing.compare
>>> matplotlib.testing.compare.comparable_formats()
['png', 'eps', 'svg', 'pdf']
  3.  pHash – The open source perceptual hash library.

To get more PDF tips and updates, subscribe to my mailing list

How to visually compare two PDF files?

I have an application written in Python, which uses the ReportLab package for exporting PDF files.

Of course, the application needs to be tested. Among other tests, the PDF export function needs to be tested to ensure that the visual rendering of PDF files did not unexpectedly change.

Since it is possible to create and save an expected-results PDF file using fabricated test data, the above implies the need to compare two PDF files. It turns out that two PDF files created from the same data at two different dates – are different, due to embedded timestamps.

Hence, the need to compare visual renderings of the PDF files. ImageMagick’s convert knows how to convert PDF files into PNG. However, one needs to set the background and remove the alpha channel.

convert knows also to perform bitwise XOR on two image files, but it must be told how to compute the bitwise XOR. This is documented in StackOverflow: Searching for a way to do Bitwise XOR on images.

The script in  https://gitlab.com/TDDPirate/compare_pdfs implements all the above.

To get more PDF tips and updates, subscribe to my mailing list

Outputting PDFs with your own fonts from your Django application

Do you use reportlab for creating PDF files from your Django application, as instructed in Outputting PDFs with Django?
Do you wish to add your own font files?
Do you need to render Hebrew text?

If yes, the following information will help you.

When installed in a virtualenv, as recommended by good working practices, reportlab searches for font files it uses in your virtualenv/lib/python3.x/site-packages/reportlab/fonts and it is not good idea to add your own font files there.

Instead, add your font files to your_project_root_directory/reportlab_extra_fonts where your_project_root_directory is where your project’s manage.py is located. Add the following to any Python script that uses reportlab (usually views.py), after all regular reportlab import and configuration statements.
# The following configures extra reportlab fonts.
import os
from django.conf import settings
reportlab.rl_config.TTFSearchPath.append(os.path.join(settings.BASE_DIR, 'reportlab_extra_fonts'))
from reportlab.pdfbase import pdfmetrics
from reportlab.pdfbase.ttfonts import TTFont
HEBREW_FONT_NAME = 'your_font_name'
pdfmetrics.registerFont(TTFont(HEBREW_FONT_NAME, 'your_font_file_name.ttf'))
# The above configures extra reportlab fonts.
If you want to properly display Hebrew in your PDF file (the probable reason why you needed to add your own fonts in the first place), you need to convert the text yourself from logical ordering to visual ordering, because reportlab (as of version 3.4.0) does not currently process BiDi text. For this purpose, install the python-bidi package in your virtualenv (using pip install python-bidi) and add the following import statement to your views.py script:
from bidi.algorithm import get_display
Now, get_display() will reorder your BiDi text.

To get more PDF tips and updates, subscribe to my mailing list

The Python module for file type identification, called ‘magic’, is not standardized

I found the hard way that the API exported by the Python module ‘magic’ differs among different versions of the module.

The version installed when installing the Debian package ‘python-magic’ expects the following API:

import magic
mymagic = magic.open(magic.MAGIC_MIME_TYPE)
mymagic.load()
mtype = mymagic.file(inpfname)
print("The MIME type of the file %s is %s" % (inpfname,mtype))

The version installed using ‘pip install python-magic’ expects the following API:

import magic
mymagic = magic.Magic(mime=True)
mtype = mymagic.from_file(inpfname)
print("The MIME type of the file %s is %s" % (inpfname,mtype))

The following code allows the rest of the script to work the same way with either version of ‘magic’:

import magic
def build_magic():
  try:
    mymagic = magic.open(magic.MAGIC_MIME_TYPE)
    mymagic.load()
  except AttributeError,e:
    mymagic = magic.Magic(mime=True)
    mymagic.file = mymagic.from_file
  return(mymagic)
mymagic = build_magic()
mtype = mymagic.file(inpfname)
print("The MIME type of the file %s is %s" % (inpfname,mtype))

PyGuile – Part 5 – Python objects (PyObjects) as proxies for Guile objects (SCMs)

An essential part of integration of Scheme (as implemented in Guile) and Python is allowing Python code to call back code implemented in Scheme. It is also desirable to be able to access data and invoke methods on otherwise-opaque objects created and managed in Scheme. The specifics of opaque object access should also be independent of the specific object system being used in the Scheme-written part of the application.

When implementing in PyGuile the proxying scheme, in which PyObjects serve as proxies for SCMs, the following need to be taken into account:

  • Data type conversion
  • Retention of references as a protection against garbage collection

Data type conversions

A Python callback can be expected to receive positional and keyword arguments, and return a result of any type. Therefore, templates (possibly trivial) for converting between PyObjects (Python data types) and SCMs (Guile data types) need to be associated with each callback.

In the case of objects, we need to associate, with each attribute getter, a template for converting the value from SCM into PyObject. With each attribute setter, a template for converting the value from PyObject into SCM needs to be associated. With each method, which can be invoked on the object, minimum two templates are needed. Three templates should be provided, in case the object needs to be manipulated by an interface, which expects both positional and keyword arguments in the object’s methods.

All the templates needed to work with a SCM (as a callback or as an object) are associated with it when it is wrapped as it is being passed from Guile to Python.

Retention of references

PyObjects, which wrap SCMs, are not expected to be seen by Guile’s garbage collector. Therefore, we need a mechanism for protecting SCMs referenced by PyObjects.

Due to efficiency considrations, Guile’s scm_permanent_object, scm_gc_protect_object or scm_gc_unprotect_object should not be used on every SCM passed to Python. The solution is to create a set object in Guile, protect it using scm_permanent_object (a single call) and then register in it all wrapped SCMs. When a wrapping PyObject’s __del__ function is invoked, one of its actions is to remove the corresponding SCM object from the set. The set will be implemented using a standard hash table, whose keys will be indexes and the data – the SCMs themselves.