CellProfiler notes

1 Warning

If you decide to upgrade libc6-dev from Debian's testing repo, be advised that it can be VERY difficult to downgrade it later. There may be a few problems with old libraries compiling on the latest libc6-dev library, so do this with caution.


2 Installation

These are my notes on installing CellProfiler on a Debian Wheezy system. I cloned the GitHub repo on June 20, 2015. The CellProfiler version is 2.1.2, rev cfb4b61.

At first, I tried but abandoned the Makefile.CP2 approach and focused instead on installing the required packages to get CellProfiler.py to run without any errors. Though I managed to get CellProfiler running, I eventually ran into problems when using it because of an error that occurred when initializing the workers. For this reason–at least I think for this reason–Cell Profiler would crash every time I ran an analysis.

I then decided to revisit the Makefile.CP2 approach. I successfully got this working on June 26, though there are still some bugs that need to be smoothed over.


2.1 Option 1: Makefile.CP2

This is the installation option that has worked the best for me, though it does require a good deal of time and effort.


2.1.1 Download required repositories

sudo apt-get install cmake libglu1-mesa-dev libhdf5-dev libmysqlclient-dev libvigraimpex3

libmysqlclient-dev is required to install the PyPi package mysql-python. If the Debian package is not present on your system, you will see a mysql_config error when trying to install it via pip.

libvigraimpex3 may or may not be required, but its Python wrappers definitely are to avoid a vigra import error while CellProfiler is running. I found information on VIGRA here.


2.1.2 Set the Java environment variables

I added the following lines to my .bashrc so the Makefile could find the Java library files.

# Setup the Java environment for Cell profiler
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-7-openjdk-amd64:/usr/lib/jvm/java-7-openjdk-amd64/include

2.1.3 Change the HOSTTYPE environment variable

At first, the Makefile complained that the HOSTTYPE environment variable wasn't set, which was not true. The real problem was that it was set to x86_64, but the Makefile expects amd64 for 64 bit Intel architectures.

I fixed this by overwriting the HOSTTYPE variable as

export HOSTTYPE="amd64"

2.1.4 Copy the Open GL library files

I already had the Mesa development libraries for Debian installed, so you may need to install these first.

Once installed, I found that there is a bug in the Make configure script for the version of wx that comes with CellProfiler. This bug causes compile-time errors when searching for the OpenGL libraries on Debian Wheezy because the configure script is not set to search the directory in which they're located. Links with more information are at the bottom of this section.

I eventually solved this problem (albeit in a dirty way) by making symlinks to the .so library files in the directory where wx would search for them.

cd /usr/lib
sudo ln -s x86_64-linux-gnu/libGL.so libGL.so
sudo ln -s x86_64-linux-gnu/libGL.so.1 libGL.so.1
sudo ln -s x86_64-linux-gnu/libGLU.a libGLU.a
sudo ln -s x86_64-linux-gnu/libGLU.so.1 libGLU.so.1
sudo ln -s x86_64-linux-gnu/libGLU.so.1.3.08005 libGLU.so.1.3.08005

For more information, see

  1. http://trac.wxwidgets.org/ticket/13375#no1
  2. http://stackoverflow.com/questions/8179139/failed-to-compile-wxwidgets-2-9-2-on-kubuntu11-1064bit-with-with-opengl-fl

2.1.5 Setup the BLAS, LAPACK, and ATLAS environments

When compiling SciPy, I ran into errors stating that my system could not find the BLAS library. These were fixed by setting the following environment variables as detailed here:

export BLAS="/usr/lib/libblas.so"
export LAPACK="/usr/lib/liblapack.so"
export ATLAS="/usr/lib/libatlas.so"

I believe that the BLAS and LAPACK libraries are provided by the libblas-dev and liblapack-dev Debian packages, so if you have problems you should check that they are installed. I do not know what package provides the ATLAS library, but it was present on my system when I looked for it.

These files might be stored elsewhere on your system. To find them, try searching for them using find:

sudo find / -name libblas.so

2.1.6 Set the correct LAPACK and ATLAS .so files

A major error I encountered using the Makefile approach occurred during the installation of cellh5:

ImportError: scipy/linalg/clapack.so: undefined symbol: clapack_sgesv

This error occurs when your Debian system is not using the right .so library file for either ATLAS, LAPACK, or both. This error was fixed for me by following these instructions; in particular, I had to change the .so file my Debian system was using for liblapack.so.3.

sudo update-alternatives --config libblas.so.3
sudo update-alternatives --config liblapack.so.3

Both of these files need to come from /usr/lib/atlas-base/atlas/ (I only needed to change the second). If you don't have this option, be sure that the Debian package libatlas3-base is installed.

Following this, the Makefile.CP2 approach successfully finished.


2.1.7 Change the src version of VIGRA to 1.10.0

Unfortunately, the vigra library shipped with Cell Profiler (version 1.7.1) does not compile on Debian Wheezy due to a ptr_diff_t error. (See this link for more information.) I tried manually fixing the afflicted files by adding a =#include <cstddef> line, but this required an enormous amount of work and ultimately led to an error that I just could not fix.

To circumvent this, I downloaded vigra 1.10.0 from their website, unzipped the tarball, renamed the folder inside it to vigra-1.7.1, then rezipped it with the name vigra-1.7.1-src.tar.gz (the same as the old tarball). This tricked the Makefile into thinking it was installing version 1.7.1 when in fact it installed vigra 1.10.0. I also deleted lines 935 and 936 from Makefile.CP2 to prevent patching the impex.hxx file. Following these steps, vigra compiled.

Note that this requires that you have already reached the point in the Makefile where vigra 1.7.1 was downloaded but failed to compile. When you receive the error, do the steps above and rerun the Makefile.


2.1.8 Upgrade libc6 and libxml2-dev

WARNING: Some suggest that upgrading libc6 or glibc carries some risk of breaking your system, though I did not encounter any problems. Downgrading these is very, very difficult, so only continue if you feel comfortable with this option. You could also just upgrade entirely to Debian Jesse.

The Wheezy stable version of libxml2-dev is too old to work with CellProfiler, which needs at least 2.9. Fortunately, 2.9 is in Debian's testing repo and Jesse, but it requires an upgraded libc6-dev

To get the testing version of libc6-dev, I first setup my pinning preferences to ensure that the security and stable repositories had the highest priority. This prevents automatically installing the new versions of everything that's in the testing repo. Based on the information found here, I did this by creating files named security.pref, stable.pref, testing.pref, unstable.pref, and experimental.pref inside the etc/apt/preferences.d directory.

Inside these files, I added:

security.pref

Package: *
Pin: release l=Debian-Security
Pin-Priority: 1000

stable.pref

Package: *
Pin: release a=stable
Pin-Priority: 995

testing.pref

Package: *
Pin: release a=testing
Pin-Priority: 750

unstable.pref

Package: *
Pin: release a=unstable
Pin-Priority: 50

experimental.pref

Package: *
Pin: release a=experimental
Pin-Priority: 1

To be safe, I also added all the lines above to one file called preferences inside the /etc/apt directory since I'm not sure if preferences or preferences.d is, well, preferable. The Debian Wiki suggests that apt uses /etc/apt/preferences, but a preferences.d folder was already present on my system. You can also set the pins for explicit packages by changing the asteriks above to specific package names and setting their pin-priority slightly above stable.

(And if you don't want to worry at all about future upgrades to packages, just remove the sources that are mentioned below from sources.list after you install libc6-dev.)

Next, I added the following lines to the end my /etc/apt/sources.list file:

# Testing repository - main, contrib and non-free branches
deb http://debian.ethz.ch/debian/ testing main non-free contrib
deb-src http://debian.ethz.ch/debian/ testing main non-free contrib



# Testing security updates repository
deb http://security.debian.org/ testing/updates main contrib non-free
deb-src http://security.debian.org/ testing/updates main contrib non-free



# Unstable repo main, contrib and non-free branches, no security updates here
deb http://debian.ethz.ch/debian/ unstable main non-free contrib
deb-src http://debian.ethz.ch/debian/ unstable main non-free contrib

I am in Switzerland so I am using the ETH mirror at http://debian.ethz.ch; you will want to change this to your local mirror. (Note that the testing security updates use a different mirror above.)

At this point, to check whether the pin-priorities were set, type

sudo apt-get update
sudo apt-cache policy libc6-dev

You should see the pin-priority numbers you set next to the various versions of the packages available at the different repos (be sure you update apt-get first). On my system, there is also a set of three asteriks next to the one that will be installed when using apt-get install.

Finally, I installed the testing version of libc6-dev with

sudo apt-get -t testing install libc6-dev

Following this, I could install libxml2-dev version 2.9 from one of the Debian non-stable repos.


2.1.9 Run the Makefile

export PREFIX="${HOME}/cp2"
export GITHOME=$PREFIX/src/CellProfiler
mkdir -p $PREFIX/src
git clone https://github.com/CellProfiler/CellProfiler $GITHOME
cd $GITHOME
make -f Makefile.CP2 PREFIX="${PREFIX}"

2.1.10 Misc. problems

Don't forget that you will have to run the Makefile and let it go to the point where VIGRA's installation fails before following the steps above to fix it.

If you get an error related to zlib.h, simply back up your local copy at /usr/include/zlib.h, and then copy the CellProfiler source zlib.h from ~/cp2/include to /usr/include.


2.1.11 Running CellProfiler

To run CellProfiler, I created a bash script containing the following lines:

#!/bin/bash

export PATH="$HOME/cp2/bin:${PATH}"
export LD_LIBRARY_PATH="$HOME/cp2/lib:$HOME/cp2/lib/mysql:$HOME/cp2/lib64:${LD_LIBRARY_PATH}"
export JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64"
export PATH="$JAVA_HOME/bin:${PATH}"
export LD_LIBRARY_PATH="${JAVA_HOME}/jre/lib/amd64:${JAVA_HOME}/jre/lib/amd64/server:${LD_LIBRARY_PATH}"

cd "$HOME/cp2/src/CellProfiler/"
python CellProfiler.py --do-not-build --do-not-fetch

I used chmod a+x my_script.sh to make the script executable and I run this script whenever I want to run CellProfiler.


2.2 Option 2: Install dependences yourself

I found this option easier than the Makefile approach, but it always results in CellProfiler crashing during an analysis, so I can not recommend it.


2.2.1 Adding wx libraries to the virtualenv

I use virtualenv's to keep my system Python environment clean. Unfortunately, wx does not play well with virtualenvs.

I fixed this problem by creating symlinks to the site-package files inside the virtualenv as suggested here. It requires that Debian's python-wxgtk2.8 package is installed.

ln -s /usr/lib/python2.7/dist-packages/wx* /home/envs/CellProfiler/lib/python2.7/site-packages/

Note that dist-packages might all be called site-packages inside your particular /usr/lib/python2.7 folder. envs is my virtual environments folder, and CellProfiler is the virtual environment I made for CellProfiler.


2.2.2 Install CellH5

I was receiving an error stating that no module named cellh5 could be found shortly after I managed to get CellProfiler running. Since CellH5 is not in PyPi, I manually installed it to my virtualenv.

pip install pandas scikit-learn lxml
git clone https://github.com/CellH5/cellh5.git
cd cellh5
~/envs/CellProfiler/bin/python setup.py install

The last step used the python binary in my CellProfiler virtualenv to ensure that cellh5 was installed locally to only that virtualenv.


2.2.3 Install vigra

I ran into some minor problems installing the Python VIGRA wrappers since I could not get them from PyPi. (I think the server that they're hosted on at the ETH in Zürich no longer hosts the software.)

Fortunately, there are wrappers in the Debian package index. I installed them to my system's site-packages with Synaptic, then copied them to my virtual env like so:

sudo apt-get install python-vigra
cp /usr/lib/pymodules/python2.7/vigra/ ~/envs/CellProfiler/lib/python2.7/site-packages/

As always, you will want to change the envs folder and CellProfiler virtualenv name to match your system. I believe you can uninstall the VIGRA wrappers from your site installation after doing this if you want to keep it clean, though I have not tried this.


2.2.4 List of installed Python packages

This is the list of Python packages installed in my CellProfiler virtualenv.

(CellProfiler)kmdouglass@kmd-laptop1:~/src/CellProfiler$ pip freeze
cellh5==1.2.0
Cython==0.22.1
h5py==2.5.0
javabridge==1.0.11
libtiff==0.4.0
lxml==3.4.4
matplotlib==1.4.3
mock==1.0.1
MySQL-python==1.2.5
nose==1.3.7
numpy==1.9.2
pandas==0.16.2
Pillow==2.8.2
pyparsing==2.0.3
python-bioformats==1.0.5
python-dateutil==2.4.2
pytz==2015.4
pyzmq==14.7.0
scikit-learn==0.16.1
scipy==0.15.1
six==1.9.0
verlib==0.1
wxPython==2.8.12.1
wxPython-common==2.8.12.1

2.2.5 CellProfiler crashes during an Analysis

Unfortunately, CellProfiler always crashes during an analysis shortly after starting all the workers with a frame error. (I am not at home now where I did most of this work and don't have the full error report).


3 Errors while running

3.1 Worker 0: ImportError: No module named readline

3.1.1 Description

When starting my first pipeline (the example http://www.cellprofiler.org/examples.shtml#HumanCells) the Worker hung because it could not find a specific Python module called readlines.

Progress Counter({u'Unprocessed': 1})
Starting workers on address tcp://127.0.0.1:62445
Worker 0: Traceback (most recent call last):
Worker 0:   File "/home/kmdouglass/cp2/src/CellProfiler/cellprofiler/analysis_wo
rker.py", line 149, in <module>                                                
Worker 0:     from cellprofiler.utilities.rpdb import Rpdb
Worker 0:   File "/home/kmdouglass/cp2/src/CellProfiler/cellprofiler/utilities/r
pdb.py", line 22, in <module>                                                  
Worker 0:     import readline  # otherwise, pdb.Pdb.__init__ hangs
Worker 0: ImportError: No module named readline

I attempted to fix this by installing libreadline-dev: sudo apt-get install libreadline-dev. However, the real problem is that the Makefile did not install a local copy of Python's readline module. I posted a bug report to their forum and am waiting on a response as of June 26, 2015.


3.1.2 Solution

I used pip to force an install of readline to the site-packages directory used by CellProfiler's Python install.

export READLINE="/home/kmdouglass/cp2/lib/python2.7/site-packages"
pip install --target=$READLINE readline