Blog

Dockerfile

The Dockerfile for this example is short and simple:

FROM tensorflow/tensorflow:latest
COPY mnist-example-for-experts.ipynb /notebooks
ENV PASSWORD test

WORKDIR "/notebooks"

CMD ["/run_jupyter.sh", "--allow-root"]

We will start with the tensorflow base image and copy in a Juypter notebook from the current directory. This notebook contains one of the TensorFlow deep learning examples. I have used it as a CPU vs. GPU benchmark in the past. We will copy it to /notebooks, a directory of Jupyter notebooks present in the base image.

The ENV PASSWORD test line sets the PASSWORD environment variable to test. This is to provide a default password to Jupyter. It should be fine for your personal machine, but you should try something more secure if you are running on an open port in the cloud. We will see such an example shortly.

The CMD directive specifies that we should execute the run_jupyter.sh script (provided by the base image). This script will run until you shut down Jupyter or the container itself.

Shell Commands

Here are the commands to build and run this workflow:

docker pull tensorflow/tensorflow

docker build -t pybay-workflow-3a .

docker run -d -p 8888:8888 --name workflow-3a-container pybay-workflow-3a:latest

In the run command, there are two differences from our prior examples:

1. We are using -d to run in detached mode.
2. The -p 8888:8888 option specifies that we want to map port 8888 in the container to port 8888 on the host.

Running the Commands

When we run the commands, we get the following output:

$docker pull tensorflow/tensorflow
Using default tag: latest
latest: Pulling from tensorflow/tensorflow
Digest: sha256:f73015504e3af53e9f747defdc2e2d857baa1140dd3cfdf92136051072502a67
Status: Image is up to date for tensorflow/tensorflow:latest
$ docker build -t pybay-workflow-3a .
Sending build context to Docker daemon  26.62kB
Step 1/5 : FROM tensorflow/tensorflow:latest
 ---> 2c8d1fd8bde4
Step 2/5 : COPY mnist-example-for-experts.ipynb /notebooks
 ---> Using cache
 ---> af4750aeab12
Step 3/5 : ENV PASSWORD test
 ---> Using cache
 ---> 615323d4b8a3
Step 4/5 : WORKDIR "/notebooks"
 ---> Using cache
 ---> fab65e003ee4
Step 5/5 : CMD ["/run_jupyter.sh", "--allow-root"]
 ---> Using cache
 ---> 250a7cabd682
Successfully built 250a7cabd682
Successfully tagged pybay-workflow-3a:latest
$docker run -d -p 8888:8888 --name workflow-3a-container pybay-workflow-3a:latest
62e246c5b8ce6c2573e5c7146ba067caeb8e920b4f67a5a8d19ec0fe536859be
$

Note that, when we execute the run command, Docker prints a long hex string (an id for the container) and then exits. The container is still running, just in the background. We can see that, by running a docker ps command (kind of like the Unix ps command for listing processes):

$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                              NAMES
62e246c5b8ce        pybay-workflow-3a:latest   "/run_jupyter.sh --a…"   2 minutes ago       Up 2 minutes        6006/tcp, 0.0.0.0:8888->8888/tcp   workflow-3a-container
$

So, we can see that it is running. If we connect our web browser to http://localhost:8888, we see a page like this:

If we enter the password we specified in the Dockerfile ("test"), we should now see a Jupyter page with a list of notebooks:

We can pick our notebook and start running it.

echo -n "Please enter a password for Jupyter:"
read -s PASSWORD
echo
echo docker run -d -p 8888:8888 --env PASSWORD=***** --name workflow-3a-container pybay-workflow-3a:latest
docker run -d -p 8888:8888 --env PASSWORD=$PASSWORD --name workflow-3a-container pybay-workflow-3a:latest

Pitfall: Container Dies Just After Startup

Perhaps you are running a detached container just like the above example, but when you go to the web browser, you get a "Can't connect to the server" error. So, you run docker ps, but see nothing:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
$

What do you do? First, you can run docker ps -a. The -a option tells Docker that you want to see a list of all container on the machine, even those not currently running. We now see our container:

$ docker ps -a
CONTAINER ID        IMAGE                      COMMAND                  CREATED              STATUS                          PORTS               NAMES
fc2e76d251f5        pybay-workflow-3a:latest   "/run_jupyter.sh --a…"   About a minute ago   Exited (1) About a minute ago                       workflow-3a-container

Ok, so we know that it exited just after startup. How do we debug this? The simplest way is to run the container in interactive mode instead of detached mode. We should see any error messages logged to the console. Here is an example:

docker run -d -p 8888:8888 --name workflow-3a-container pybay-workflow-3a:latest
The Jupyter HTML Notebook.

This launches a Tornado based HTML Notebook Server that serves up an
HTML5/Javascript Notebook client.
...
[C 18:26:27.008 NotebookApp] Bad config encountered during initialization:
[C 18:26:27.008 NotebookApp] Unrecognized flag: '--allowroot'
$

From this, I can tell that the Jupyter startup script was passed a bad option: --allowroot instead of allow-root. I can fix this and rebuild my container.

$ docker exec -it workflow-3a-container /bin/bash
root@9fd5d385cf80:/notebooks# pwd
/notebooks
root@9fd5d385cf80:/notebooks# ls
1_hello_tensorflow.ipynb  2_getting_started.ipynb  3_mnist_from_scratch.ipynb  BUILD  LICENSE  mnist-example-for-experts.ipynb
root@9fd5d385cf80:/notebooks# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 18:30 ?        00:00:00 bash /run_jupyter.sh --allow-root
root         7     1  0 18:30 ?        00:00:00 /usr/bin/python /usr/local/bin/jupyter-notebook --allow-root
root        31     0  0 18:33 pts/0    00:00:00 /bin/bash
root        41    31  0 18:33 pts/0    00:00:00 ps -ef
root@9fd5d385cf80:/notebooks# exit
$

Pitfall: GPU Access

Perhaps you tried running this TensorFlow notebook on a machine with one or more GPUs. Since TensorFlow supports GPU execution, it should just use the GPU, right? No. It is not quite that simple!

By default, Docker does not see the GPU installed on your machine. There are a few things you need to do first to run your containers on your Linux box with GPU access:

1. Install the Nvidia GPU driver for Linux. You will need the actual proprietary version from Nvidia, the open source ones will not work for this use case.
2. Install nvidia-docker, a Nvidia plugin for Docker, using the instructions at https://github.com/NVIDIA/nvidia-docker
3. Pick a base image that has the CUDA libraries and other dependencies for your framework (e.g. tensorflow/tensorflow:latest-gpu)

This is only supported on Linux - Nvidia does not support Docker on Mac or Windows.

In the example code, workflow-3b adapts workflow 3 to run on a GPU.

Although this is a few more steps than running only on the CPU, I have found that it is still much easier than dealing with the GPU library dependencies yourself.

Shell Commands

Automating this scenario is a little more complex than the previous ones, but still can be done in a 40 line shell script. This script may be found in the Git repo at workflow-4/run.sh.

We start by pulling the image from Docker Hub and recreating some directories:

docker pull neo4j:3.4

rm -rf ./data ./logs
mkdir ./data
mkdir ./logs

Now we need to do a little shell magic to get the names of the CSV files we will import and adjust their paths to be under /imports (as they will be seen within the container). The result of this step is the definition of two environment variables, $NODES_ARGS containing the importer command line arguments for node files and $EDGES_ARGS containing the importer command line arguments for edge files. Here is the bash code:

pushd  ./graph-import-files
for f in nodes-*.csv
  do
      echo "Processing $f"
      NODES_ARGS="$NODES_ARGS --nodes /imports/$f"
 done
  for f in edges-*_to_*.csv
  do
      echo "Processing $f"
      EDGES_ARGS="$EDGES_ARGS --relationships /imports/$f"
  done
popd

Now, we can run the first container to create the database from the CSV files:

docker run -it --rm \
   --volume=`pwd`/data:/data \
   --volume=`pwd`/graph-import-files:/imports \
   --volume=`pwd`/logs:/logs \
   --env=NEO4J_AUTH=neo4j/test \
   neo4j:3.4 bin/neo4j-admin import $NODES_ARGS $EDGES_ARGS

Finally, we can create the database server container:

docker run -d \
   --volume=`pwd`/data:/data \
   --volume=`pwd`/logs:/logs \
   --env=NEO4J_AUTH=neo4j/test \
   -p 7474:7474 -p 7687:7687 \
   --name pybay-neo4j neo4j:3.4

The Neo4j browser based query UI is on port 7474 and the database client/server connections happen on port 7687. If we connect our browser to http://localhost:7474, and enter the username and password (specified via environment variables as neo4j/test), we should be able to log into the running database. We can enter the query match (n) return n; at the prompt to display all the nodes of our graph. It should look something like this: