benchmarks-two-photon-imaging.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:af66061fafaf0d5131c93de42f191a112ab640e0b373fc68fa68a44c249b4085"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "<img style='float: left' width=\"150px\" src=\"http://thunder-project.github.io/images/logo-outline.svg\">\n",
      "<br>\n",
      "<br>\n",
      "<br>\n",
      "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Benchmarking loading and analysis in <a href='http://thunder-project.github.io/'><font color='#EB8B34'>Thunder</font></a>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This notebook shows the performance of basic loading, data manipulation, and analysis routines in Thunder on two-photon calcium imaging data. \n",
      "\n",
      "**Environment**\n",
      "- Run on a private HPC with 10 nodes, each node had 128 GB RAM and 16 cores (comparable to CC2.8xlarge)\n",
      "- Spark version 1.2.0, Thunder version 0.5.0, Python version 2.7.6 \n",
      "\n",
      "**Data**\n",
      "- Data are from mouse two-photo imaging experiments in the Svoboda lab\n",
      "- Data dimensions are 512 x 512 x 4 x 9000 with a size on disk of 20GB as images with `uint16` precision.\n",
      "- Data available at [*location on S3*]"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Import neccessary libraries"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import timeit\n",
      "from thunder import PCA, KMeans"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Specify the data location. Change as needed when running on EC2 or on a private networked file system."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "path = '/tier2/freeman/Freeman/mouse.svoboda/tactile.navigation/1/'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>\n",
      "Image loading"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = tsc.loadImages(path + 'images-binary', dims=[512,512,4], recursive=True, ext='bin')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 3.12 s per loop\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>\n",
      "Basic image manipulations"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = tsc.loadImages(path + 'images-binary', dims=[512,512,4], recursive=True, ext='bin')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.medianFilter(3).count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 10.7 s per loop\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.gaussianFilter(3).count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 5.45 s per loop\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.crop(minbound=(0,0,0),maxbound=(100,100,2)).count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 3.11 s per loop\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>\n",
      "Conversion to Series"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = tsc.loadImages(path + 'images-binary', dims=[512,512,4], recursive=True, ext='bin')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.toSeries().count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 53.2 s per loop\n"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.saveAsBinarySeries(path + 'series', overwrite=True)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 52 s per loop\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>\n",
      "Series loading"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = tsc.loadSeries(path + 'series')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 3.12 s per loop\n"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>\n",
      "Series manipulations"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = tsc.loadSeries(path + 'series')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.zscore().count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 3.8 s per loop\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.seriesMean().count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 3.53 s per loop\n"
       ]
      }
     ],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.seriesMean().pack()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 7.06 s per loop\n"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.toTimeSeries().detrend('linear').count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 11.6 s per loop\n"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>\n",
      "PCA"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = tsc.loadSeries(path + 'series')\n",
      "data.cache()\n",
      "data.count();"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 model = PCA(k=3, svdMethod='em').fit(data)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 2min 19s per loop\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>\n",
      "Cross-correlation"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = tsc.loadSeries(path + 'series')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 data.toSpatialSeries().localCorr(2).pack()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 2min 13s per loop\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "<hr>\n",
      "KMeans"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data = tsc.loadSeries(path + 'series')\n",
      "data.cache()\n",
      "data.count();"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%timeit -n 3 KMeans(k=5).fit(data)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 loops, best of 3: 1min 40s per loop\n"
       ]
      }
     ],
     "prompt_number": 7
    }
   ],
   "metadata": {}
  }
 ]
}