OMERO Notes

Table of Contents

1 The Open Microscopy Environment - OMERO

I have been dabbling with the Open Microscopy Environment, a suite of open-source software for the storage and manipulation of microscopy data.

I'm especially interested in OMERO, which provides client-server software for image storage and handling of metadata. I'm interested in this because I believe that it could improve our lab's data provenance, i.e. the record-keeping that describes how our results are derived from our raw data. This is extremely important, yet extremely difficult to achieve when working with the enormous data sets that we normally do.

On this page I've maintained notes on my attempts to setup a working OMERO server.


2 OMERO Virtual Appliance

The OMERO virtual appliance is a virtual system that allows you to experiment with the OMERO server before actually installing one. This sounds nice, but is proving to be a bit tricky to get working.


2.1 Setup 1

This section describes my first attempt to setup the OMERO Virtual Appliance.


2.1.1 My computer system

  • Dell Precision T7500
  • Windows Server 2008 R2 Enterprise w/Service Pack 1, 64 bit
  • Intel Xeon X5690 @ 3.47 GHz (2 processors)
  • 48.0 GB RAM

2.1.2 Directions for Installing the OMERO Virtual Appliance

I followed the directions on OMERO's webpage for this process (https://www.openmicroscopy.org/site/support/omero4/users/virtual-appliance.html).

Software versions

  • VirtualBox version 4.3.8 r92456 for Windows (VirtualBox-4.3.8-92456Win)
  • OMERO Virtual Appliance version 5.0.0-b19 (OMERO.vm-5.0.0-b19.ova)

During the VirtualBox install I accepted all the defaults. This caused no problems.

I also loaded the virtual appliance into Virtual Box and verified that Port Forwarding was setup as required. It was.


2.1.3 Problem: Loading and booting the virtual appliance

I attempted to boot the virtual appliance as it is described in the OMERO documentation. After powering on the virtual machine, I see the following screens, in this order:

  1. The Oracle/VirtualBox splash screen.
  2. A black window with the lines "Grub loading." and "Welcome to GRUB!". The second line appears roughly two seconds after the first appears.
  3. A new window with the line "Booting 'Debian GNU/Linux, with Linux 3.2.0-4-amd64'" at top. The lines "Loading Linux 3.2.0-4-amd64 …" and "Loading initial ramdisk …" appear after this sequentially.

Unfortunately, at this point, the virtual machine screen goes black with only a non-blinking cursor. I let the machine sit for about half an hour and saw no change from this screen. I also could not log onto the OMERO.web interface. or with the OMERO.insight client.

setup1-vbScreen.jpg

The VirtualBox screen immediately before it goes blank in Setup 1.

2.1.4 Solution: Enabled Virtualization Support in the BIOS

I solved the above problem by enabling virtualization support in my computer's BIOS. (I don't know why this works. I need to read about it.) To do this on my Dell T7500, I followed the following steps:

  1. Restart my computer, press at the splash screen, and enter the system configuration tool
  2. Select the "Virtualization Support" folder on the left side of the screen.
  3. Select the "Virtualization" tab.
  4. Check the box labeled "Enable Intel Virtualization Technology".
  5. Select the "VT for Direction I/O" tab.
  6. Check "Enable Intel VT for Direct I/O."

Save the settings and restart the computer. The OMERO Virtual Appliance should now run in VirtualBox.


2.2 Setup 2

Setup 2 is essentially the same as Setup 1 but with the virtualization settings enabled in the BIOS.


2.2.1 Problem: The OMERO Virtual Appliance hangs after half an hour

After testing the Virtual Appliance for roughly half an hour I suddently lost connection to the server in the OMERO.insight client.

I ALT-TABed to the appliance's terminal window to see that it had resumed, the blank, hanging state indicative of the problem I encountered in Setup 1.

I should perhaps allocate more memory to the Virtual Appliance in Virtual Box.


3 OMERO Test Server Installation

Before installing the OMERO on our dedicated server, I would like to walk through the installation on a different machine running Windows Server 2008. This is a desktop machine that I also use as a work station, but it should emulate well any problems I might encounter with the actual OMERO install.

I will be following the OMERO.server Windows Installation guide found here: https://www.openmicroscopy.org/site/support/omero5/sysadmins/windows/server-installation.html


3.1 Hardware

The hardware for this test machine is described in the section My computer system above. It is the same system that I ran the virtual appliance on.


3.2 Software prequisites

According to the installation guide, I need to download and install the following software. All software should be 32 bit or 64 bit, i.e. I should not mix and match. Also, I initially had Python (x,y) 2.7.5 installed. Before attempting the OMERO.server prequisites, I uninstalled it from my computer.

I installed the following software in the order found in this list. For some reason, Ice is discussed before the installation of Python in the OMERO.server documentation, but it requires Python. Therefore, I chose to install it after Python.

  1. Java SE Development Kit (JDK) – JDK 6 and above are supported.
  2. Python 2.6 (not 2.7 or later!) – 64 bit Python has not been fully tested, though it should work. Ice in Windows won't work with Python 2.7.
  3. Python for Windows extensions (PyWin32) – Check the OMERO.server installation guide for the proper version.
  4. Additional Python libraries – See the OMERO.server installtion guide.
  5. Ice (3.3.x or 3.4.x) – Ice 3.5 is not supported in the Windows install.
  6. PostgreSQL 8.4 (or later)

3.2.1 Java SE Development Kit (JDK) 7u51

I installed JDK 7u51 for 64 bit Windows. The download link for jdk-7u51-windows-x64.exe is here: http://www.oracle.com/technetwork/java/javase/downloads/index.html

I accepted all default options, including installing it to C:\Program Files\Java\jdk1.7.0_51.


3.2.2 Python 2.6.6

Python, besides being my choice scripting language, is also required for Ice to work (see Ice 3.4.2). I wanted to start fresh like I would be doing on the actual server, so I uninstalled my Python(x,y) package. I then downloaded the 64 bit Windows pre-compiled binary for Python 2.6.6 from https://www.python.org/download/releases/2.6.6/ and installed it. The file name is python-2.6.6.amd64.msi.


  1. Python for Windows Build 218

    I downloaded and installed pywin32-218.5.win-amd64-py2.6.exe from http://www.lfd.uci.edu/~gohlke/pythonlibs/.


  2. Edit the environment variables

    At this point, I edited my System Path environment variable. I added "C:\Python26" to the beginning of the variable.

    To edit it, I right-clicked on Computer from the Start Menu and selected Properties… In this menu, I selected Advanced system settings from this menu. Under the Advanced tab, I clicked the button labeled Environment Variables… and edited the System variable. Make sure that you don't edit the User variable.


  3. pip 1.5.4

    I nexted installed pip 1.5.4, the Python package index. (Pillow is easily installed with pip, and I will likely use pip in the future anyway.) To do this, I downloaded get-pip.py from http://www.pip-installer.org/en/latest/installing.html.

    I navigated to the directory where I downloaded the file in the Windows command prompt and executed the command python get-pip.py


  4. (OLD) Fixing non-ascii encoding errors in Python 2.7

    NOTE I performed this step only during my first installation when I attempted to install OMERO with Python 2.7. Since ICE can't use Python 2.7, and mimetypes.py is different in Python 2.6, this step isn't necessary when using Python 2.6.

    I work in French-speaking Switzerland, which means the registry on my Windows workstation often contains characters that are not found in the ascii encoding. This became apparent when I attempted to install Pillow using pip. pip's easy_install.py script threw an error when accessing mimetypes.py in the Python library. This bug is documented here: http://bugs.python.org/issue16617

    1. Install GnuWin Patch 2.5.9 from here http://gnuwin32.sourceforge.net/packages/patch.htm.
    2. Download the mimetypes.patch file from the Python bug report mentioned above. Place it in the same directory as mimetypes.py. For me, this was C:\Python27\Lib\.
    3. Move the patch file into the same directory as mimetypes.py.
    4. In a command prompt with root privileges, run the command patch < mimetypes.patch.
      1. Note that I needed to add the directory containing the patch binary file to my environment variables. This was C:\Program Files (x86)\GnuWin32\bin.
      2. Note that the patch changes two lines in mimetypes.py, but the second line that the patch refers to in mimetypes.py in my install is wrong. I manually changed line 272 (not 270!) from except UnicodeEncodeError: to except UnicodeError.

  5. Pillow 2.3.1

    Pillow-2.3.1.zip from https://pypi.python.org/pypi/Pillow/2.3.1

    To install Pillow on this Windows machine, I followed these directions: http://pillow.readthedocs.org/en/latest/installation.html#windows-installation.

    In brief, I opened a command prompt with administrator privileges and switched to the directory containing the pip binaries.For me the directory is C:\Python26\Scripts. I ran the command easy_install Pillow.

    This originally throws an error about a missing file called vcvarsall.bat because I have Visual Studio 2012 installed (not 2008). I changed the VS90COMNTOOLS environment variable by running the command set VS90COMNTOOLS=%VS110COMNTOOLS%. See http://stackoverflow.com/questions/2817869/error-unable-to-find-vcvarsall-bat.

    My next attempt to install Pillow produced the error:

    build\temp.win-amd64-2.6\Release\_imaging.pyd.manifest : general
    error c1010070: Failed to load and Parse the manifest. The system
    cannot find the file specified./
    

    This issue is addressed here: http://stackoverflow.com/questions/12418735/pip-fails-to-install-pil-or-pillow-with-mt-exe-error and here: http://bugs.python.org/issue4431. I fixed it by adding ld_args.append('/MANIFEST') to below the line ld_args.append('/MANIFESTFILE:' + temp_manifest) in msvc9compiler.py, which is found in C:\Python26\Lib\disutils\.

    Pillow succesfully installed after this.


  6. Numpy1.8.0, Scipy-0.14.0b1, and matplotlib 1.3.1

    Because I installed 64 bit Python, I needed to download and install the unofficial 64 bit binaries of Numpy and Scipy found at the links below.

    I installed the packages in the above list in top-to-bottom order.


  7. numexpr 2.1, Cython 0.20.1, HDF4 1.8.12, and PyTables 3.1

    I attempted to install PyTables 3.1 using pip via the command easy_install tables, but I received an error informing me that I needed numexpr 2.0.0 or greater.

    I downloaded numexpr-2.3.1.win-amd64-py2.6.exe from http://www.lfd.uci.edu/~gohlke/pythonlibs/#numexpr and installed numexpr. For some reason the pip package wouldn't install.

    Next, I executed easy_install numexpr, which added numexpr 2.1 to the easy-install.pth file. However, PyTables now needed Cython, so I executed the command pip install Cython to install Cython (version 0.20.1 as of March 19, 2014).

    Following this, I learned that an HDF5 install is ALSO required for PyTables. (Managing package dependencies on Windows sucks.) So I downloaded hdf5-1.8.12-win64-vs10shared.zip from http://www.hdfgroup.org/HDF5/release/obtain5.html, unzipped it, and ran the executable it contained. Be sure to add HDF5 to the PATH for all users during the install.

    This unfortunately didn't fix the problem of the unrecognized HDF5 library, even when I added the HDF5 directory to the HDF5_DIR environment variable. However, I learned that the PyTables binaries come prepackaged with the required HDF5 dll's; I only need to install numexpr and Cython. I learned this here: http://pytables.github.io/usersguide/installation.html#binaryinstallationdescr

    So, I downloaded tables-3.1.0.win-amd64-py2.6.exe from http://www.lfd.uci.edu/~gohlke/pythonlibs/#pytables and ran it. Beautiful.

    Finally, I enabled LZO compression in PyTables because they mentioned that it may speed up some operations. I downloaded lzo-2.03-win64.zip from http://sourceforge.net/projects/pytables/files/lzo-win/. I copied lzo2.dll found within it to C:\Windows\System32.

    Inside the Python shell, I ran the following commands:

    import tables
    tables.test()
    

    After 223.588 seconds, the test stopped and reported that I had a working copy of PyTables.


3.2.3 Coffee

At this point I went to get coffee.


3.2.4 Ice 3.4.2

Ice 3.5.0 does not work with an OMERO Windows install. Instead, I downloaded and installed Ice 3.4.2 from http://zeroc.com/download_3_4_2.html. The file name is Ice-3.4.2.msi.


3.2.5 PostgreSQL 9.3.3

I downloaded the postgresql-9.3.3-1-windows-x64.exe from http://www.enterprisedb.com/products-services-training/pgdownload#windows.

During the graphical install I set a super-user password and selected the default port of 5432. I set the locale to US English since my lab works in English, despite being located in French-speaking Switzerland.


3.2.6 Setting up the environment variables

I followed the OMERO install guide's directions for setting the environment variables. Specifically, I

  • added "C:\Program Files (x86)\ZeroC\Ice-3.4.2\bin\x64" to the end of the System PATH variable;
  • created a new system environment variable called PYTHONPATH and set its value to "C:\Program Files (x86)\ZeroC\Ice-3.4.2\python\x64".

Be sure to use the "x64" folders if you've been installing 64 bit programs!


3.2.7 Computer restart

At this point, at the recommendation of the OMERO install guide, I restarted the computer.


3.2.8 Database creation

I followed the steps in the OMERO installation guide exactly to create a database using postgre's pgAdmin III program.


3.3 OMERO.server software installation

I downloaded the Ice 3.4 Server from OMERO's downloads section, found here: http://downloads.openmicroscopy.org/omero/5.0.0/

I downloaded the 3.4 version because 3.5 does not yet work on a Windows install due to incompatibility with Python 3.


3.3.1 Initial problems with the omero batch file and Python 2.7

The default install directory for Ice was in C:\Program Files (x86). Unfortunately, this creates issues in the batch file setpythonpath.bat that's called by the omero.bat script.

To fix it, I placed double quotes around %PYTHONPATH% and %%i after PARSE on line 11 of setpythonpath.bat.

However, I then received a notice that the OMERO configuration could not find a dll with the OMERO.cli module. According to the OMERO troubleshooting page (https://www.openmicroscopy.org/site/support/omero5/sysadmins/troubleshooting.html), this usually occurs when there are issues with PYTHONPATH.

I wondered if the issue could be with the spaces in C:\Program Files (x86), so I uninstalled Ice and reinstalled it to C:\ZeroC\Ice-3.4.2. I updated the appropriate environment variables to reflect this change and tried the install again.

Installing to a directory without spaces fixed the batch file errors, but the configuration file still cannot find OMERO.cli.


  1. Ice requires Python 2.6

    After asking on the forum, I discovered that Ice 3.4 requires Python 2.6, NOT Python 2.7. This is presumably causing the issues with the missing DLL. I misread the OMERO install documentation, which read Python 2 (2.6 or later). The forum discusssion is here: https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7466

    I updated all the preceding notes to reflect a Python 2.6 installation.


3.3.2 Second install attempt with Python 2.6

Once I reinstalled all the prereqs associated with Python 2.6, I opened a command prompt with administrator rights and navigated to C:\OMERO.server. I unzipped the OMERO server files into this directory. I also created an empty directory named C:\OMERO.

In the prompt I entered the following commands, with MYUSER and MYPASS replaced with the appropriate information.

bin\omero config set omero.db.name omero_database
bin\omero config set omero.db.user MYUSER
bin\omero config set omero.db.pass MYPASS
bin\omero config set omero.data.dir C:\\OMERO

This time the script ran without any output, as opposed to the OMERO.cli errors I received when I attempted to configure OMERO with Python 2.7 installed.


3.3.3 Fixing issues with Java and PostgreSQL not being on your path

I next entered the following commands in a command prompt with administrator privileges. My goal was to create the database initialization script, but the command returned the following error concerning Java.

C:\Windows\system32>cd C:\OMERO.server

C:\OMERO.server>bin\omero db script
Traceback (most recent call last):
  File "C:\OMERO.server\bin\\omero", line 125, in <module>
    rv = omero.cli.argv()
  File "C:\OMERO.server\lib\python\omero\cli.py", line 1195, in argv
    cli.invoke(args[1:])
  File "C:\OMERO.server\lib\python\omero\cli.py", line 745, in invoke
    stop = self.onecmd(line, previous_args)
  File "C:\OMERO.server\lib\python\omero\cli.py", line 814, in onecmd
    self.execute(line, previous_args)
  File "C:\OMERO.server\lib\python\omero\cli.py", line 894, in execute
    args.func(args)
  File "C:\OMERO.server\lib\python\omero\plugins\db.py", line 258, in script
    data = self.ctx.initData({})
  File "C:\OMERO.server\lib\python\omero\cli.py", line 1018, in initData
    output = getprefs(["get"], str(path(self._cwd(None)) / "lib"))
  File "C:\OMERO.server\lib\python\omero\plugins\prefs.py", line 55, in getprefs

    return omero.java.run(cmd, chdir=dir)
  File "C:\OMERO.server\lib\python\omero\java.py", line 100, in run
    check_java(command)
  File "C:\OMERO.server\lib\python\omero\java.py", line 28, in check_java
    raise Exception("Java could not be found. (Executable=%s)" % command[0])
Exception: Java could not be found. (Executable=java)

I also noticed that I could not run java from the command prompt:

C:\OMERO.server>java -version
'java' is not recognized as an internal or external command,
operable program or batch file.

The output of the OMERO diagnostic tool was:

C:\OMERO.server>bin\omero admin diagnostics

================================================================================

OMERO Diagnostics 5.0.0-ice34-b19
================================================================================


Commands:   java -version                  not found
Commands:   python -V                      2.6.6     (C:\Python26\python.EXE)
Commands:   icegridnode --version          3.4.2     (C:\ZeroC\Ice-3.4.2\bin\x64
\icegridnode.EXE)
Commands:   icegridadmin --version         3.4.2     (C:\ZeroC\Ice-3.4.2\bin\x64
\icegridadmin.EXE)
Commands:   psql --version                 not found

Server:     icegridnode                    not started

Cannot list deployed applications.

Log dir:    C:\OMERO.server\var\log        exists

Log files:  Blitz-0.log                    n/a
Log files:  DropBox.log                    n/a
Log files:  FileServer.log                 n/a
Log files:  Indexer-0.log                  n/a
Log files:  MonitorServer.log              n/a
Log files:  OMEROweb.log                   n/a
Log files:  Processor-0.log                n/a
Log files:  Tables-0.log                   n/a
Log files:  TestDropBox.log                n/a
Log files:  Total size                     0.00 MB

(There were some errors on log parsing as well, but I'll save those for later.) The key thing to note here is that neither Java nor PostgreSQL are recognized by OMERO.

After poking around the web, I came across this old post to the OMERO listserv: http://lists.openmicroscopy.org.uk/pipermail/ome-users/2012-November/003415.html. Oscar solved this problem by installing the 32 bit JDK instead of the 64 bit version. Instead of installing the 32 bit JDK, I searched for locations of java.exe on my computer and found the binaries stored in C:\Windows\SysWOW64. This folder is NOT on my Windows PATH system environment variable, so I added it. Restarting the command prompt and running the OMERO diagnostics command like above now reports that Java is found.

As for PostgreSQL, I found this OMERO forum post: http://www.openmicroscopy.org/community/viewtopic.php?f=5&t=304. I added C:\Program Files\PostgreSQL\9.3\bin to my PATH system environment variable, restarted the command prompt, and saw that OMERO now recognized Java and PostgreSQL.

C:\OMERO.server>bin\omero admin diagnostics

================================================================================

OMERO Diagnostics 5.0.0-ice34-b19
================================================================================


Commands:   java -version                  1.7.0     (C:\Windows\SysWOW64\java.E
XE)
Commands:   python -V                      2.6.6     (C:\Python26\python.EXE)
Commands:   icegridnode --version          3.4.2     (C:\ZeroC\Ice-3.4.2\bin\x64
\icegridnode.EXE)
Commands:   icegridadmin --version         3.4.2     (C:\ZeroC\Ice-3.4.2\bin\x64
\icegridadmin.EXE)
Commands:   psql --version                 9.3.3     (C:\Program Files\PostgreSQ
L\9.3\bin\psql.EXE)

Server:     icegridnode                    not started

Cannot list deployed applications.

Log dir:    C:\OMERO.server\var\log        exists

Log files:  Blitz-0.log                    n/a
Log files:  DropBox.log                    n/a
Log files:  FileServer.log                 n/a
Log files:  Indexer-0.log                  n/a
Log files:  MonitorServer.log              n/a
Log files:  OMEROweb.log                   n/a
Log files:  Processor-0.log                n/a
Log files:  Tables-0.log                   n/a
Log files:  TestDropBox.log                n/a
Log files:  Total size                     0.00 MB

3.3.4 Generating the SQL initialization script

After fixing the issues with the PATH variable, I successfully generated the SQL script as instructed in the OMERO documentation.

C:\OMERO.server>bin\omero db script
Please enter omero.db.version [OMERO5.0]:
Please enter omero.db.patch [0]:
Please enter password for OMERO root user:
Please re-enter password for OMERO root user:
Saving to C:\OMERO.server\OMERO5.0__0.sql

3.3.5 Database initialization

This worked just as described in the documentation. Be sure to enter the username for the user that you created in pgAdmin III, NOT the default postgres user.


3.3.6 Error when starting the server and solution

With everything setup, I attempted to start the server and received this error:

C:\OMERO.server>bin\omero admin start
Found default value: c:\omero_dist\var\master
Attempting to correct...
Converting from c:\omero_dist to C:\OMERO.server
Changes made: 6
Creating C:\OMERO.server\var\master
Creating C:\OMERO.server\var\registry
No descriptor given. Using etc\grid\windefault.xml
Installing OMERO.master Windows service.
Successfully installed OMERO.master Windows service.
Starting OMERO.master Windows service.
Waiting on startup. Use CTRL-C to exit
.............................
Failed to startup some components after 300 seconds
Calling "stop" on remaining components
OMERO.master service deleted.
Waiting on shutdown. Use CTRL-C to exit
......

I made a quick hop over to the relevant section of the Troubleshooting page at https://www.openmicroscopy.org/site/support/omero5/sysadmins/troubleshooting.html#server-fails-to-start.

I verified both that PostgreSQL was configured in the manner suggested in the documentation and that I had the proper permissions on C:\OMERO.

I then remembered that I changed the password to the PostgreSQL user that can access the database because I forgot the initial one :) So I reran the OMERO config commands and successfully started the server after that.


3.3.7 Logging in for the first time

To login, I executed the following commands with the proper password:

C:\OMERO.server>bin\omero login
Server: [localhost]
Username: [douglass]root
Password:
Created session 1f039a39-b29e-4744-9c95-491407052c11 (root@localhost:4064). Idle
 timeout: 10.0 min. Current group: system

I also tested whether I could login with the OMERO.insight client. This works as well.

Note you MUST be logged into the PostgreSQL console as well. I made this mistake and didn't do it the first time. See my forum post here: https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7468


3.3.8 Increasing the Java Virtual Machine memory

Following the documentation, I next wanted to test whether increasing the memory available to the Java virtual machine caused any problems.

On line 192 of C:\OMERO.server\etc\grid\templates.xml I changed -Xmx512M to -Xmx2048M. On line 201, I changed XX:MaxPermSize=128m to XX:MaxPermSize=256M. I'm not sure whether the change from a lowercase 'm' to an uppercase 'M' matters, but I made it anyway according to the documentation.

The server failed to restart with these settings. But, by returning them to their original settings, the server was able to start. The documentation does state that increasing the memory size might prevent the server from starting on some test systems, so I presume that once I have the software installed on the real server it shouldn't be an issue.

Also note that changing -Xmx512M to -Xmx1204M, as well as increasing XX:MaxPermSize to 256M will work on my machine, so I don't necessarily have to stick to the recommended values in the documentation.


3.3.9 OMERO.web and administration

I followed the directions in the documentation exactly to configure the Django web server and ran it. After receiving a very long string of outputs from the console, the web server ran. (Be sure that the OMERO server is running in the background as well, something that I did not do the first time.)

Going to http://localhost:4080 led me to the OMERO.web login screen. I noticed that I could not log onto the web server with the root user account. However, I could log on to the server with the OMERO.insight client and through the console. This might be a limitation of the internal Django web server and not an actual problem with the install.

When I would try to log in as root, I'd see the following lines in my command prompt window:

[21/Mar/2014 16:15:19] "POST /webclient/login/?url=%2Fwebclient%2F HTTP/1.1" 302
 0
[21/Mar/2014 16:15:19] "GET /webclient/ HTTP/1.1" 302 0
[21/Mar/2014 16:15:19] "GET /webclient/login/?url=%2Fwebclient%2F HTTP/1.1" 200
3358

I'm not sure what this means, but for the moment I'm willing to accept this.

UPDATE: April 3, 2014

After speaking with the OMERO development team, I learned how to access the OMERO.web development server from the host machine.

I performed the following steps with the OMERO server (but not the web server) running. Enter the following commands into a command prompt with administrator privileges:

bin\omero config set omero.web.session_engine
bin\omero config set omero.web.cache_backend
bin\omero admin restart

Now, you should be able to log into the OMERO.web interface with root or any other user you may have created.


3.4 Summary of software installation

Before proceeding further, I'd like to highlight several things I learned about the software installation. I have not yet tried doing any actual work with the OMERO clients or setup backups and security. The OMERO server software is just simply running.

  • Make sure you have the correct versions of the software. Python 2.7 will not work with Ice 3.4.
  • Installation of the Python prerequisite packages was the most difficult part and required some hacking of special environment variables and Python files that interact with Visual Studio. These are not related to OMERO itself, but OMERO does rely on them for some of their functionality. I hope that my notes will help with their installation on a Windows machine.
  • 64 bit versions of the various software packages like Java and Postgre may not be in directories that are already in the Windows PATH system environment variable. You should add them manually.
  • The OMERO team is very active on the help forums and mailing list. Don't be afraid to use them!

4 OMERO Test Server Maintenance

Now that I have the server software up and running, there are a number of things I want to do first on this test system before installing OMERO on our dedicated server and putting it into production. A short list of these things includes:

  1. Understand the concept of programs running in the foreground/background in Windows
  2. Deploy OMERO.web on an IIS platform
  3. Set up routine backups in Windows
  4. Read up on the system security page found here: http://www.openmicroscopy.org/site/support/omero5/sysadmins/server-security.html
  5. Set up OMERO.dropbox

I will explore these items and post my notes as I do so. Once I feel comfortable with them, I'll go ahead with the production server install.


4.1 Run the OMERO server in the background

On Windows Server machines, there are two ways to run programs in the "background." Though I don't fully understand the difference between the two, some Googling around has led me to believe that services run very close to the hardware level and are difficult to create because they require .NET programming or 3rd party software. Tasks, on the other hand, are easy to schedule using Windows Task Scheduler and are often the most appropriate choice when deciding between services and tasks. (See for example this page: http://weblogs.asp.net/jgalloway/archive/2005/10/24/428303.aspx)

Here's how I managed to start the OMERO server everytime the server is booted without the need to login as a user in Windows 2008.

  1. Open the Server Manager by either clicking the icon in the Windows Taskbar or by selecting its shortcut in Start -> Administrative Tools.
  2. In the left-most pane you should see a tree of options. The top-most option is Server Manager with the computer name in parantheses. Under this level select Configuration -> Task Scheduler.

omero_taskScheduler_1.JPG

The Windows Task Scheduler screen.
  1. In the right-most pane, select Create Basic Task….
  2. Type in a name and description for the task. I named mine OMERO.start. Click the Next button.
  3. Select the When the computer starts radio button to run the OMERO batch script when the computer boots. Click the Next button.
  4. Select the Start a program radio button and click Next.
  5. In the text box, enter the full path and file name of the batch file, C:\OMERO.server\bin\omero.bat. In the Add arguments (optional) text box, enter admin start. In the Start in (optional) text box, enter C:\OMERO.server\bin. Click next.

omero_taskScheduler_2.JPG

Settings for selecting the OMERO batch script.
  1. In the final pane, check the Open the Properties dialog for this task when I click Finish box and click the Finish button.
  2. In the Properties window under the General tab, check the Run whether user is logged on or not radio button and check Run with highest privileges.

You can now click the OK button and the OMERO server should startup at run time.

Note that earlier I discovered that I needed postgreSQL to be running to startup the OMERO server, but I've now discovered that this is not true. I don't know why there is a difference, but upon rebooting, the OMERO server started without problems. I could log on with the OMERO.insight client did not have to schedule a postgreSQL task.

What is also interesting is that I can see a running service named postgresql-x64-9.3 in the Windows Task Manager, so perhaps PostgreSQL runs in the background after a full reboot.


4.2 Deploying OMERO.web on IIS

IIS was already setup on my Windows 2008 machine. It seems relatively easy to add if it's not on your machine, however.

The OMERO.web setup documentation is found here: https://www.openmicroscopy.org/site/support/omero5/sysadmins/windows/install-web.html. I first set the web application server to the recommended FastCGI TCP by entering the following command in the command prompt with administrator rights:

bin\omero config set omero.web.application_server "fastcgi-tcp"

I next verified that ISAPI was enabled in IIS by going into the Server Manager application, selecting Roles -> Web Server (IIS) in the left-hand pane, then scrolling down to Role Services in the Web Server (IIS) right-hand pane. On my system, ISAPI Extensions and ISAPI Filters are already installed.

omero_webServer.JPG

Screenshot of the Server Manager screen showing installed ISAPI Extensions and Filters.

Continuing through the steps in the documentation, I next navigated to http://code.google.com/p/isapi-wsgi/downloads/list and downloaded isapi_wsgi-0.4.2.win32.exe. I ran the executable but it reported that no Python install was found in the registry.

After reading the Google Code ISAPI-WSGI page (http://code.google.com/p/isapi-wsgi/wiki/InstallationInstructions) I next tried to install isapi_wsgi through pip instead. In the command prompt I executed the following commands:

cd C:\Python26\Scripts
easy_install isapi_wsgi

This seemed to successfully install the ISAPI WSGI package onto my system.

The next step involved enabling 32-bit applications in IIS, since I'm running a 64 bit version of Windows Server 2008. To do this, I navigated to Start -> Administrative Tools -> Internet Information Services (IIS) Manager. In the left-hand pane, I selected my server machine, then clicked Application Pools. In the right-hand pane, I right-clicked DefaultAppPool and selected Advanced Settings…. Then, I set the Enable 32-Bit Applications flag to True.

omero_webServer_2.JPG

Screenshot showing where you can enable 32-bit applications in IIS.

Finally, I entered the final three commands recommended in the documentation:

C:\omero_dist>bin\omero config set omero.web.session_engine "django.contrib.sessions.backends.cache"

C:\omero_dist>bin\omero config set omero.web.cache_backend "file://C:/windows/temp/"

C:\omero_dist>bin\omero web iis

After struggling a bit to understand why I could not access OMERO.web from my browser, I realized I skipped a step. I had to turn on IIS 6 backwards compatibility.

To do this, I navigated to the Server Manager and selected Roles in the left-hand pane. In the right-hand pane, I scrolled down to the Web Server (IIS) section and clicked Add Role Services on the right. From here I installed IIS 6 WMI Compatibility and IIS 6 Scripting Tools. (Note that IIS 6 Metabase Compatibility was already installed.)

OMERO.web still would not start after this. I also tried accessing the web interface by navigating to http://localhost/omero but this returned an "Error 500-Internal server error." With little other idea what was happening, I turned to the forums and eventually found a solution with the help of someone else having similar problems. The two forum topics are below. I also found a related thread on the listserv which I believe is related and included it below:

  1. http://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7476&p=13822#p13822
  2. http://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7486
  3. http://lists.openmicroscopy.org.uk/pipermail/ome-users/2014-April/004322.html

The solution to the problem has two parts: clearing the troublesome config variables and disabling 32 bit support in IIS. First, I followed the steps that the OMERO team recommended to unset the two configuration variables that were causing problems with logging into the development server.

bin\omero config set omero.web.session_engine
bin\omero config set omero.web.cache_backend
bin\omero admin restart

Next, I disabled 32-bit support in the IIS Manager window. Specifically, I went to Start -> Administrator Tools -> Internet Information Services (IIS) Manager. I clicked my server in the left-hand pane, then selected "Application Pools" beneath it. I right-clicked DefaultAppPool and selected "Advanced Settings…" from the menu. In the new menu, I disabled the 32 bit support. This is in contradiction to the OMERO server documentation as of April 4, 2014, but since they're in the process of updating their documentation with the release of OMERO 5.0, it might have already been addressed.

This allowed me to access the web server with my browser without having to restart IIS, OMERO, or even run the bin\omero web start command. The address of the server is localhost\/omero. I can also now access the OMERO.web interface from other computers on my network by going to a browser and entering my server's IP address followed by \/omero.


4.2.1 Summary of IIS web deployment

To deploy OMERO.web on April 4, 2014 using the same software versions that I'm using, you will have to clear the omero.web.session_engine and omero.web.cache_backend configuration variables and disable 32 bit support in Windows IIS manager. With these two changes, the OMERO server documenation should lead to a working OMERO server.


4.3 Setting up routine backups

Note: Windows Task Scheduler is the Windows equivalent to Unix's cron, which should be the tool to use to routinely backup the various parts of OMERO.

The OMERO documentation recommends frequent backups of 1) the PostgreSQL database and 2) the OMERO binary data. To backup the PostgreSQL database, there is a pre-made Windows batch file for doing just this at the PostgreSQL Wiki found here (though I found that it should be modified): https://wiki.postgresql.org/wiki/Automated_Backup_on_Windows. This batch file can be run atuomatically using Windows Task Scheduler.

I first created a new directory for backups in C:\Program Files\PostgreSQL\9.3\backup\. In this directory, I created a batch file called OMERO_backup.bat that contained the following:

@echo off
echo Date is %date%

set DB_NAME=omero_database
set BACKUP_FILE=%DB_NAME%_%date%.pg_dump
echo backup file name is %BACKUP_FILE%
SET PGPASSWORD=<PASSWORD>
echo on
..\bin\pg_dump -i -h localhost -p 5432 -U <DBUSER> -F c -f %BACKUP_FILE% %DB_NAME%

@echo off
set PGPASSWORD=

Obviously, you should replace <PASSWORD> and <DBUSER> with the appropriate information, which in this case is the user you chose as the owner of omero_database. Also, make sure that only administrator's have access to the folder that contains this batch file since the PostgreSQL password is saved as plain text.

(In case you're wondering what the pg_dump command does, or about PostgreSQL backups in general, there is a very nice and clear explanation here: http://www.postgresql.org/docs/9.1/interactive/backup.html)

Finally, this batch file can be set to run daily using Windows Task Scheduler in a manner similar to the one used to start OMERO at the server startup.

Backing up the OMERO binary data store comes down to deciding on proper backup software. The documentation recommends using the open source rdiff-backup, but the website for this software suggests that it's not well-tested in Windows. Despite this, I tried out the Windows binaries for rdiff-backup version 1.2.8 anyway (they are found here: http://www.nongnu.org/rdiff-backup/). I did this because we've had issues restoring Windows backup compressed files in the past. As a result, my lab switched to using SyncBackFree, but this doesn't run on a Windows Server machine. In the absence of other easy options, I decided to stick with rdiff-backup.

After downloading the rdiff-backup zip file, I unzipped it and placed the executable in C:\Windows\, though any directory in the PATH environement variable would work. I then created a batch file called OMERO_binary_backup.bat that contained the following lines:

@echo off
SET FROM=C:\OMERO
SET TO=N:\OMERO_backup

rdiff-backup %FROM% %TO%

I scheduled this batch file to run daily as well.


4.4 Notes on security

I read through the security page at http://www.openmicroscopy.org/site/support/omero5/sysadmins/server-security.html to get an idea about security in OMERO.

It would seem that I have a relatively simple job at manging security on this system since I am not hosting the binary repository on a different server than the OMERO install. If instead the binaries were on a different server, or if I was using LDAP to synchornize user accounts in multiple programs including OMERO, then I should take additional measures to ensure that the LDAP communications were secure/encrypted.

I found good introductory resources explaining SSL (the mechanism used for signal encryption) and LDAP (the uniform language for data access across different clients and servers) at these two sites, respecitvely:

  1. http://www.webopedia.com/TERM/S/SSL.html
  2. http://www.gracion.com/server/whatldap.html

In addition, I read up on packet filtering. Below, I've provided links to useful resources. The first is an introductory explanation of the concept of packet filtering, followed by the man-page for iptables (only for Linux, but still informative). The last resource is the Windows Server 2008 documenation for adding packet filters to your server:

  1. http://searchnetworking.techtarget.com/definition/packet-filtering
  2. http://ipset.netfilter.org/iptables.man.html
  3. http://technet.microsoft.com/en-us/library/cc732746(v=ws.10).aspx

On the Windows doc page, it recommends using either static packet filters or Windows Firewall, but not both. I discovered that I can add rules for incoming ports in Windows Firewall instead of performing static packet filtering.

Finally, if need be, I can restrict clients' rights to turn off SSL encryption for non-password-related communications with the OMERO server. Clients might do this to speed up their data access rate with the server. Instructions on how to do this are listed in the OMERO security docs on the page mentioned above.


4.5 DropBox

OMERO.dropbox is described here: http://www.openmicroscopy.org/site/support/omero5/sysadmins/dropbox.html

It doesn't appear too difficult to setup. I only need to add a few folders to C:\OMERO\DropBox and set a few parameters in etc\grid\templates.xml. I'll wait to set this up on the actual server.


4.6 Summary of Test Server Maintenance

Most of my time went into deploying OMERO.web onto IIS on the test server. This was because of a bug, either in the software or documentation, that asked that we enable 32-bit compatibility when in fact it prevented OMERO.web from displaying in a web browser.

The other tasks, like setting recurring backups, are relatively simple, but still very important.


5 OMERO Server Installation

At this point I've been able to run OMERO successfully in Windows Server 2008, access the server from other computers on our network, and play around inside OMERO.insight as the root user.

I'm now ready to install the software onto our full-time server.


5.1 Server Specs

  • Transtec Calleo 4260
  • 2x Intel Xeon E5-2620v2 processors
  • 14.5 TB total storage space
  • 64 GB memory
  • Windows 2012 R2 Server 64 bit

The biggest difference with this machine is that it will be running Windows 2012 Server, not Windows 2008 Server.


5.2 Pre-installation

5.2.1 Turn off Windows Installer RDS Compatibility

Before installing anything, I first had to turn off Windows Installer compatibility with remote desktop. This is because Windows uses a utility that queues installations in case there are multiple administrators working on a system. Unfortunately, programs like the Java SDK that call multiple installers will hang during installation on a remote desktop session due to the queuing.

I turned off RDS compatibility by entering Edit group policy in the Start menu search bar. Then, in the window that appears, I navigated to Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Application Compatibility\Turn off Windows Installer RDS Compatibility

After double-clicking Turn off Windows Installer RDS Compatibility, a new window popped up in which I clicked on the Enabled radio button.

ATTENTION: You may want to re-enable this when you're done in case you have multiple administrators. See http://www.gearbytes.com/2010/11/looping-windows-installer-coordinator/ for more information.

5.3 Software

Similar to my test install, I installed the following software in the order listed below.

  • Java SE Development Kit 7u51 (64-bit)
  • Python 2.6.6 (64-bit)
    • Python for Windows Build 218
    • pip 1.5.4
    • *Pillow 2.4.0
    • Numpy 1.8.0
    • Scipy-0.14.0b1
    • matplotlib 1.3.1
    • numexpr 2.1
    • Cython 0.20.1
    • HDF4 1.8.12
    • Pytables 3.1
  • Ice 3.4.2
  • PostgreSQL 9.3.3
  • Denotes a difference from the Windows Server 2008 install.

5.4 Differences from test server installation

There were the following differences in installing software between the Windows Server 2012 and Windows Server 2008 machine:

  1. The Python Pillow package installs using the command easy_install Pillow in Windows Server 2012. It installed Pillow 2.4.0 for Python 2.6 (64-bit).
  2. Microsoft Visual Studio 2013 is installed on this new machine, so I set the environment variable set VS90COMNTOOLS=%VS120COMNTOOLS%.
  3. I installed OMERO.server 5.0.1, not 5.0.0.
  4. I had to add C:\Program Files\PostgreSQL\9.3\bin to my PATH system environment variable to get the bin\omero admin diagnostics command to recognize psql.
  5. When scheduling a Windows task to automatically run the OMERO server at startup, I ensured that the task DID NOT stop if it ran for longer than three days.
  6. I placed rdiff-backup.exe in C:\Windows\SysWOW64. (See this forum topic: http://stackoverflow.com/questions/16760449/py2exe-in-c-windows-system32-on-64-bit-windows)
  7. I'm backing up the OMERO repository and SQL database to two different directories, C:\OMERO_backup and C:\OMERO_SQL_backup.
  8. I created two batch files for server management that are run daily.
    OMERO_backup.bat
    Backup the SQL data base using PostgreSQL dumps
    OMERO_rdiff_backup.bat
    Backup the repository and SQL dump
  9. omero admin start is scheduled to run everytime the server starts.
  10. I opened ports 4064 and 4063 in the firewall to incoming TCP traffic. Before I did this, I could not remotely connect to the server.
  11. IMPORTANT Be sure not to schedule the SQL dumps and the rdiff-backup of the SQL dumps at the same time. Otherwise, you may not back up the latest file. You'll want to schedule rdiff-backups slight after the SQL dump.

6 OMERO Server Administration

This section is not meant to be a walkthrough or tutorial, but merely a place to keep notes on how I've administered our server.


6.1 Group and User Creation

I created a group for our lab that had "Private" permissions. In the future, once we're done testing the software, we'll probably upgrade the permissions to "Read-only."

According to the OMERO documentation, administrators can't severe Read-only permissions on data if they're initially set to Read-only. However, a group can be promoted to read-only from private. Additionally, a group can be switched between Read-only and "Read-annotate" quite easily.


6.2 Requirements for OMERO.movie

The OMERO.movie script generates a movie from a file containing a z-stack, time series, or multiple color channels. However, it did not work without installing mencoder first. To do this, I simply downloaded and unpacked the latest mplayer software for Windows. I added the directory containing mencoder.exe to my system PATH variable and restarted the OMERO server. After this, I could run the OMERO.movie script.

http://www.openmicroscopy.org/site/support/omero4/sysadmins/omeromovie.html