Getting uniqe docker container name from within container

I had a use case in which I needed to obtain a unique identifier of a container from within the container. The problem is that container is started with docker-compose using scale option so that a number of indentical containers are started. Googling around gave me some options like examining /proc/self/cgroups, but in my case, this file didn't contain anything useful. Another solution was to pass environment variable from the outside which also looked like a mess. In the end, I came up with not so elegant solution, but the one that works. I use Python, but the concept could be used with other programming languages as well.

First, environment variable contains image name, e.g. crawler. So, you can use the following Python expression to obtain this name:

import os
hostname = os.environ['HOSTNAME']

Variable hostname will contain crawler in our example. Next, resolve this name to IP address:

import socket
ip = socket.gethostbyname(hostname)

This will give you IP address of the container, e.g. 172.25.0.12. Now, you should do a reverse lookup. BUT, you can not use gethostbyaddr since it will return to you the same hostname as the one you obtained from HOSTNAME environment variable. The reason is that gethostbyname and gethostbyaddr use /etc/hosts file first. Instead, you have to use DNS. Unfortunately, this is where external library is needed. You can install and use dnspython, and then do a reverse lookup:

from dns import resolver,reversename
addr=reversename.from_address(ip)
str(resolver.resolve(addr,"PTR")[0]).split('.')[0]

The reason for the last mess is that resolver.resolve returns tupple with FQDN in the zero-th element, and then FQDN has to be splitted on dots to take the hostname only.

RSA parameters in PEM file...

If you've ever read anything about how RSA works you most probably know that RSA is based on arithmetic modulo some large integer. First, let N be a product of two, very large, primes p and q. N is n-bit number, those days minimally 1024 bits, and thus p and q are half that size, so that after multiplication we get required number of bits. Next, there are two numbers e and d that satisfy relation $eq = 1\ mod\ \phi(N)$, where $\phi(N)=(p-1)(q-1)$. $(N, e)$ is encryption key - or what's frequently referred to as a public key, while $(N,d)$ is decryption key, or a private key. The question now is: Given a PEM file with private key, how to find out those parameters for a specific public/private keys? In this post I'll show three ways to do it. The first two are using tools that are a standard part of OpenSSL/GnuTLS libraries. The third one is using Python.

OpenSSL/GnuTLS

Those two, especially OpenSSL, are very popular and complex cryptograhpic libraries. Both of them come with a very capable command line tools. In the case of OpenSSL the there is a single binary, openssl, that performs its function depending on the second argument. In this post, I'll use arguments rsa and genrsa that manipulate and generate, respectively, RSA keys. GnuTLS, on the other hand, has a several tools, of which we'll use certtool.

Now, first thing is to create RSA public/private key. You can do this using the following commands:

openssl genrsa -out key.pem 512

or, if you use GnuTLS, then:

certtool --generate-privkey --bits 512 --outfile key.pem

In both cases the generated RSA modulus (N) will have 512 bits, and the keys will be written into output file key.pem. You should note few things here:

1. The output file contains both private and public keys!
2. We didn't encrypt output file, which is recommended to do.
3. 512 bits these days is a way insecure. Use minimally 1024 bits.

Ok, now we can request information about this RSA keys. Again, you can do that using openssl in the following way:

openssl rsa -in key.pem -noout -text

or, using GnuTLS:

certtool -k --infile key.pem

In both cases you'll receive the following output (GnuTLS is a bit more verbose):

Private-Key: (512 bit)
modulus:
    00:ba:b6:78:3b:1c:15:f1:d9:e3:48:16:5e:e7:8e:
    fd:a0:9d:2f:ee:1b:b8:9b:3d:d3:ea:f4:ad:fb:1b:
    6e:ef:b2:b5:cd:ee:38:e9:f8:6d:64:c9:ea:95:ae:
    87:13:5a:23:8b:2f:0b:e8:bb:c6:f8:c6:c4:ee:64:
    3c:d4:97:bd:a3
publicExponent: 65537 (0x10001)
privateExponent:
    39:6b:07:ca:55:b6:c1:eb:59:a3:bf:8d:6b:f4:63:
    36:d3:5f:fb:ff:76:63:f7:3d:86:51:bc:77:2e:56:
    8d:4b:87:73:e0:53:bd:17:e8:4a:e8:df:f5:86:14:
    65:60:f2:4f:03:02:3b:e9:23:c6:d3:ce:b3:1d:e9:
    13:1a:0f:b1
prime1:
    00:d6:b1:f9:53:8c:56:96:79:c0:bd:68:6c:b9:07:
    e7:9c:70:de:f5:61:ed:bb:51:12:1d:24:37:0f:cc:
    bf:8a:95
prime2:
    00:de:a2:52:be:a1:a4:eb:d7:48:24:95:c5:2c:05:
    bd:5f:7f:74:d5:12:bd:7c:5f:f1:8e:45:a2:50:26:
    ec:d1:57
exponent1:
    66:61:30:58:1b:10:1f:69:a7:f3:aa:9c:4e:0f:ea:
    ee:bb:14:57:47:7f:aa:57:9a:9f:b2:e9:5e:eb:70:
    5b:91
exponent2:
    22:2d:8f:40:5e:b6:5f:d2:5b:eb:e9:e6:2c:1c:f1:
    76:90:ad:91:ec:5f:94:91:72:16:e2:4f:c9:b8:40:
    10:df
coefficient:
    22:9c:f3:1f:85:68:a3:36:ab:07:87:ed:a4:c0:e5:
    ef:13:a8:28:02:55:35:c1:76:96:86:97:58:08:90:
    6e:70

In this output we see parameters N (modulus), e (publicExponent), d (privateExponent), p (prime1), and q (prime2). Also, what's written in there are values d mod (p-1) (exponent1),  d mod (q-1) (exponent2) and q^-1 mod p (coefficient).

The interesting thing to note is the value of publicExponent, it is 65537. Namely, the size of public exponent isn't so important and by having such a low value it is relatively easy to encrypt messages (rising to this exponent requires 17 multiplications). This is very frequent value for encryption key, i.e. public exponent. privateExponent, on the other hand, has to be large and random, so that it isn't easy to guess.

Python

The recommended library to use to manipulate RSA keys is m2crypto which is a wrapper to OpenSSL library. There are many other libraries, of course, and you can find an older comparison here. Unfortunately, I was unable to find downloadable version on the Internet.

Anyway, to be able to manipulate RSA keys you have to import RSA module from M2Crypto, i.e.:

from M2Crypto import RSA

Then, to load RSA private/public key from a file use the following line:

rsa=RSA.load_key('key.pem')

You can also generate new RSA private key as follows:

rsa=RSA.gen_key(512, 65537)

In this case you are generating private key based on public key 65537 (we saw that this is very frequently used public key) and we require 512 bit modulus. You can find out modulus length using Python's len() function, i.e.:

>>> len(rsa)
512

To obtain N, you can inquire attribute n of the object holding RSA key, i.e.:

>>> rsa.n

To obtain pair (e,d), or public key, you can use the following method:

>>> RSA.pub()

Alternatively, you can find out only e in the following way:

>>> rsa.e

As far as I could see, there is no way to find other parameters besides N, e and bit length.

F17 and Python setuptools mixup...

For some time now I was having problems with Python's setuptools package. When I downloaded some package and tried to install it with the usual:

python setup.py install

I would receive the following error:

Traceback (most recent call last):
  File "setup.py", line 1, in
    from setuptools import setup, find_packages
  File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 2, in
    from setuptools.extension import Extension, Library
  File "/usr/lib/python2.7/site-packages/setuptools/extension.py", line 5, in
    from setuptools.dist import _get_unpatched
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 6, in
    from setuptools.command.install import install
  File "/usr/lib/python2.7/site-packages/setuptools/command/__init__.py", line 8, in
    from setuptools.command import install_scripts
  File "/usr/lib/python2.7/site-packages/setuptools/command/install_scripts.py", line 3, in
    from pkg_resources import Distribution, PathMetadata, ensure_directory
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2755, in
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 704, in subscribe
    callback(dist)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2755, in
    add_activation_listener(lambda dist: dist.activate())
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2258, in activate
    map(declare_namespace, self._get_metadata('namespace_packages.txt'))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1847, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1817, in _handle_ns
    loader.load_module(packageName); module.__path__ = path
  File "/usr/lib64/python2.7/pkgutil.py", line 246, in load_module
    mod = imp.load_module(fullname, self.file, self.filename, self.etc)
  File "/usr/lib/python2.7/site-packages/chameleon/__init__.py", line 1, in
    from .zpt.template import PageTemplate
  File "/usr/lib/python2.7/site-packages/chameleon/zpt/template.py", line 9, in
    from ..i18n import fast_translate
ImportError: cannot import name fast_translate

I already managed to solve this problem some time ago, but it reappeared and it was driving me nuts! And I forgot what I did! Googling didn't tell me a lot. So, I had to look into the code.

I had to search modules on disk, use print to see which one is actually loaded, and rpm/yum to query packages, but to cut the story short, in the end I found out that I had some dangling files and directories, probably because I did upgrade instead of clean install.The following ones were offending ones:

/usr/lib/python2.7/site-packages/chameleon/ast
/usr/lib/python2.7/site-packages/chameleon/core
/usr/lib/python2.7/site-packages/chameleon/genshi
/usr/lib/python2.7/site-packages/chameleon/i18n

This is easy to verify. Using 'rpm -qf <dir_or_file' shows for all of them (including files in those directories) that they are not owned by any package. So, I simply removed then and suddenly everything worked!

Fetching book data via ISBN...

Well, I needed a way to obtain book data based on its ISBN. Additional constraint was that it had to be in Python. Search quickly revealed that there are multiple Web services, one of which is ISBNdb. But there are others too. Look at this StackExchange question for some tips.

So, I decided to go with ISBNdb because I quickly stumbled on Python module for it. Additionally, I realized that Calibre uses this library, too. This turned out to be a mistake. I lost more than hour trying to make it work, just for a single search by ISBN. Googling for some example code didn't bring a lot. So, here is what I learned, maybe it helps someone out there.

First, let me say that there is a bug in the code in subversion. If you checked out version 41, then the bug is certainly there. You need to open file keys.py and change line 22 that reads like this:

_KEYS.append(k)

into:

 _KEYS[key] = k

Next, you also have to change line 62 in the same line that reads like this:

def __new__(cls, key, force_=False):

into this:

def __new__(cls, key, name, force_=False):

Also, to use this library, and ISBNdb in general, you have to register in order to get a key. Key is a string of 8 characters.

Now, we are ready to try the library. The easiest way is to position yourself into directory that contains isbndb/ directory from subversion (that way Python interpreter will be able to find the module, otherwise, you need to move it into some "system" place, or manipulate sys.path array.

The commands are:

>>> import isbndb

>>> isbndb.addKey('12345678')
Got from isbndb:
<?xml version="1.0" encoding="UTF-8"?>
<ISBNdb server_time="2012-08-21T14:16:45Z">
<KeyStats granted="9" access_key="12345678" requests="9" limit="0" /&gt:
<BookList total_results="0" page_size="10" page_number="1" shown_results="0"></BookList>
</ISBNdb>

True
>>> books=isbndb.IsbndbSearch(searchValue='0071378596')

>>> book=books[0]

>>> book.Title
'Mastering technical mathematics'

And that's enough to get you started. The rest you can find by youself using Python's introspection features and some code browsing.

In the end, I think that this library isn't mature enough, it also has debug statements embedded into it and the documentation is poor, at best!

Searching for packet catpuring and interface manipulation library for Python...

I needed a script that would monitor network traffic and capture and process only DHCP traffic. It turned out I couldn't find such script so I decided to write one (more about that script in another post). For a language I decided to use Python. That was the easy part. Now, I had to decide which libraries I will use that will allow me to capture network traffic, decode DHCP request and responses, and manipulate IP addresses on interfaces.

I started with the network traffic capturing. pcap library is the library for network capture, so it was natural for me to search for a Python interface to this library. I found several such interfaces, i.e. pcap, pylibpcap, pypcap, and pcapy. There is also library interface specifically for Python 3, i.e. py3kcap. While searching for pcap interface, three other Python libraries poped out: libdnet (here is the old project page), dpkt and scapy.

But, not all libraries are equal, nor they serve the same purpose. libdnet allows sending packets, manipulation with kernel's routing tables, firewall and arp cache. So, besides Ethernet and IP, it doesn't offer much more in term of supported protocols. dpkt, on the other hand, is made just for this purpose! It supports easy creation and parsing of different TCP/IP protocols. Finally, Scapy is a swiss army knife of network manipulation. It offers shell in which one can manipulate packets, but also can be used within other scripts. Unfortunately, while browsing the source of Scapy I realized that it uses os.popen interface and calls external programs. So, this actually was enough for me to eliminate scapy from further consideration.

The next elimination criteria is availability of the packages within CentOS and Fedora. I try to hold on prepackaged software as much as possible, so quick search (yum search) showed that on both, CentOS 6 and Fedora 17, there are packages for pcapy and dpkt (named python-dpkt). For some reason, there is dnet, but python interface isn't packaged. I found this bugzilla entry, but without any answer!

So, I settled on pcapy and dpkt. The only piece of puzzle that was missing now is how to manipulate interface addresses. I stumbled on netifaces, which allows me to obtain information about interfaces and also on this post for Windows. But all the results I got were on how to obtain IP address. In the end, I gave up and decided that I'll try to use libdnet even though I'll have to compile it from the source. Either that, or I'll use raw sockets and ioctls which are accessible from Python using standard libraries.

And for the end, as a curiosity, I'll mention that there is Python interface to IPTables, python-iptables, which is also packaged for Fedora.