save multiple tif images instead of one

[ObadaM]

I am trying to save my corrected tif image after I applied a cross correlation on it. with totif() function.

I am using coalesce to be only one file but it is not working.

corrected.coalesce(1).totif('corrected','tif',False)

am I doing something wrong ?

[boazmohar]

hi @ObadaM, sorry for the late response.
totif() will write one tif per record (corrected.shape[0]) and not per partition, so coalesce(), will not change the number of files that it will write.
see http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD for coalesce.

What is the shape of your corrected images object?
Maybe a data.reduce(lambda x, y: np.dstack((x, y))).totif() would work, but I am not sure.

[ObadaM]

hi @boazmohar Thanks for your reply and sorry for the late reply.

I tried your suggestion and I got the following error

Traceback (most recent call last):
  File "/home/l-01-314/pycharm/pycharm-2016.1.3/helpers/pydev/pydevd.py", line 1531, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/l-01-314/pycharm/pycharm-2016.1.3/helpers/pydev/pydevd.py", line 938, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/l-01-314/pycharm/pycharm-2016.1.3/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/l-01-314/PycharmProjects/test/main.py", line 51, in <module>
    corrected.totif('cor/','corr.tif',True)
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/images.py", line 503, in totif
    totif(self, path, prefix=prefix, overwrite=overwrite)
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/writers.py", line 56, in totif
    images.foreach(lambda x: writer.write(tobuffer(x)))
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/images.py", line 145, in foreach
    [func(kv) for kv in enumerate(self.values)]
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/images.py", line 145, in <listcomp>
    [func(kv) for kv in enumerate(self.values)]
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/writers.py", line 56, in <lambda>
    images.foreach(lambda x: writer.write(tobuffer(x)))
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/writers.py", line 52, in tobuffer
    imsave(bytebuf, img, format='TIFF')
  File "/usr/anaconda3/lib/python3.5/site-packages/scipy/misc/pilutil.py", line 195, in imsave
    im = toimage(arr, channel_axis=2)
  File "/usr/anaconda3/lib/python3.5/site-packages/scipy/misc/pilutil.py", line 287, in toimage
    raise ValueError("'arr' does not have a suitable array shape for "
ValueError: 'arr' does not have a suitable array shape for any mode.

The result image from the reduce was a tif image with only one frame.

[boazmohar]

@ObadaM Hi, It is hard for me to understand the error without knowing the code that generated it.
Especially because Spark is lazy It would throw an error very late compared where the problem is in the code.

It looks like something went wrong before the writing to tif.
Could you try:

1. doing .first().shape on the original data you load up
2. on corrected.first().shape
3. and corrected.reduce(lambda x, y: np.dstack((x, y))).first().shape

If none of this produces an error and all the shapes make sense, we could investigated deeper about the writing to tif step.

[ObadaM]

Here is my code, the shape is the same always. I am new to python and I find problem solving this kind of issues.

img = td.images.fromtif(path1,'tif',None,None,False,1,10,None,sc,None)
print (img.first().shape)

img = img.subsample(2)
print (img.first().shape)

ref = img[0:frames,:,:]
print (ref.first().shape)

import Motion_Correction as MC
corrected = MC.Thunder_MotionCorrection(img,ref)
print (corrected.first().shape)

corrected = corrected.reduce(lambda x, y: np.dstack((x, y)))
print (corrected.first().shape)
corrected.totif('cor/','corr.tif',True)

(768, 1024)
(384, 512)
(384, 512)
(384, 512)
(384, 512, 810)

Traceback (most recent call last):
  File "/home/l-01-314/pycharm/pycharm-2016.1.3/helpers/pydev/pydevd.py", line 1531, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/l-01-314/pycharm/pycharm-2016.1.3/helpers/pydev/pydevd.py", line 938, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/l-01-314/pycharm/pycharm-2016.1.3/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/l-01-314/PycharmProjects/test/main.py", line 74, in <module>
    corrected.totif('cor/','corr.tif',True)
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/images.py", line 503, in totif
    totif(self, path, prefix=prefix, overwrite=overwrite)
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/writers.py", line 56, in totif
    images.foreach(lambda x: writer.write(tobuffer(x)))
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/images.py", line 145, in foreach
    [func(kv) for kv in enumerate(self.values)]
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/images.py", line 145, in <listcomp>
    [func(kv) for kv in enumerate(self.values)]
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/writers.py", line 56, in <lambda>
    images.foreach(lambda x: writer.write(tobuffer(x)))
  File "/usr/anaconda3/lib/python3.5/site-packages/thunder/images/writers.py", line 52, in tobuffer
    imsave(bytebuf, img, format='TIFF')
  File "/usr/anaconda3/lib/python3.5/site-packages/scipy/misc/pilutil.py", line 195, in imsave
    im = toimage(arr, channel_axis=2)
  File "/usr/anaconda3/lib/python3.5/site-packages/scipy/misc/pilutil.py", line 287, in toimage
    raise ValueError("'arr' does not have a suitable array shape for "
ValueError: 'arr' does not have a suitable array shape for any mode.

Thanks for your help!

[boazmohar]

@ObadaM, It seems that the function used to save tif files in Thunder doesn't support 3d images.
see: http://docs.scipy.org/doc/scipy/reference/generated/scipy.misc.imsave.html
What you can do is to to collect the result locally and use tifffile from scikit-image, it also comes from thunder:

import thunder.rdds.fileio.tifffile as tifffile
local = corrected.toarray()
writer1 = tifffile.TiffWriter('cor/corr.tif')
writer1.save(local.transpose((2, 0, 1)).astype(dtype='float32'), photometric="minisblack")
writer1.close()
del writer1

@freeman-lab Is there a reason not to switch to tifffile for saving as well?

[ObadaM]

@boazmohar I could not find rdds in thunder. i have the last version of thunder 1.0.0
are you sure from the import ?

Traceback (most recent call last):
  File "/home/l-01-314/PycharmProjects/test/main.py", line 110, in <module>
    import thunder.rdds.fileio.tifffile as tifffile
ImportError: No module named 'thunder.rdds'

[boazmohar]

Sorry about the import.
Try to import from scikit-image.

Thanks,
Boaz On Fri, Jun 17, 2016 at 10:18 AM, ObadaM < notifications@github.com [notifications@github.com] > wrote:
@boazmohar [https://github.com/boazmohar] I could not find rdds in thunder. i have the last version of thunder 1.0.0
are you sure from the import ?

Traceback (most recent call last):
File "/home/l-01-314/PycharmProjects/test/main.py", line 110, in
import thunder.rdds.fileio.tifffile as tifffile
ImportError: No module named 'thunder.rdds'

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub [https://github.com//issues/328#issuecomment-226698319] , or mute the thread [https://github.com/notifications/unsubscribe/AI_zmR971S4AU3kaLxWPT1e9yDKEDl5Cks5qMkpSgaJpZM4ItUic] .

[ObadaM]

it is okay but also skimage does not have this import.