Optimise dicom processing

[mr-martian]

Originally posted by @jonorthwash at 2018-07-10T04:36:17Z (original issue)

• Use multiple threads to load dicom faster.
• Clean up variables pixels, arr, img after processed (these eat a certain amount of memory).

[mr-martian]

Originally posted by @BartholomaeusAstrum at 2018-07-18T20:52:32Z

This is the profile of the dicom loading function currently:

Total time: 23.3962 s
File: UltraTrace.py
Function: process at line 1618

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
  1618                                           	@profile
  1619                                           	def process(self):
  1620                                           		'''
  1621                                           		perform the dicom->PNG operation
  1622                                           		'''
  1623         1         32.0     32.0      0.0  		print( 'Reading DICOM data ...', end='\r' )
  1624                                           
  1625         1         16.0     16.0      0.0  		if self.isLoaded == False:
  1626         1          1.0      1.0      0.0  			try:
  1627         1          9.0      9.0      0.0  				dicomfile = self.app.Data.getFileLevel( '.dicom' )
  1628         1    4454541.0 4454541.0     19.0  				self.dicom = dicom.read_file( dicomfile )
  1629                                           
  1630                                           			except dicom.errors.InvalidDicomError:
  1631                                           				print( 'Unable to read DICOM file: %s' % dicomfile )
  1632                                           				return False
  1633                                           
  1634         1        701.0    701.0      0.0  		pixels = self.dicom.pixel_array # np.array
  1635                                           
  1636                                           		# check encoding, manipulate array if we need to
  1637         1          2.0      2.0      0.0  		if len(pixels.shape) == 3:		# greyscale
  1638         1          1.0      1.0      0.0  			RGB = False
  1639         1          1.0      1.0      0.0  			frames, rows, columns = pixels.shape
  1640                                           		elif len(pixels.shape) == 4:	# RGB-encoded
  1641                                           			RGB = True
  1642                                           			if pixels.shape[0] == 3:	# handle RGB-first
  1643                                           				rgb, frames, rows, columns = pixels.shape
  1644                                           			else:						# and RGB-last
  1645                                           				frames, rows, columns, rgb = pixels.shape
  1646                                           			pixels = pixels.reshape([ frames, rows, columns, rgb ])
  1647                                           
  1648         1          1.0      1.0      0.0  		processedData = {}
  1649                                           
  1650                                           		# write to a special directory
  1651         1          2.0      2.0      0.0  		outputpath = os.path.join(
  1652         1          5.0      5.0      0.0  			self.app.Data.getTopLevel('path'),
  1653         1         20.0     20.0      0.0  			self.app.Data.getFileLevel('name')+'_dicom_to_png' )
  1654         1         30.0     30.0      0.0  		if os.path.exists( outputpath ) == False:
  1655         1      62257.0  62257.0      0.3  			os.mkdir( outputpath )
  1656                                           
  1657                                           		# grab one frame at a time to write (and provide a progress indicator)
  1658         1        115.0    115.0      0.0  		printProgressBar(0, frames, prefix = 'Processing:', suffix = 'complete')
  1659       729       1157.0      1.6      0.0  		for f in range( frames ):
  1660                                           
  1661       728      49878.0     68.5      0.2  			printProgressBar(f+1, frames, prefix = 'Processing:', suffix = ('complete (%d of %d)' % (f+1,frames)))
  1662                                           
  1663       728       7152.0      9.8      0.0  			arr = pixels[ f,:,:,: ] if RGB else pixels[ f,:,: ]
  1664       728      80843.0    111.0      0.3  			img = Image.fromarray( arr )
  1665                                           
  1666       728      10820.0     14.9      0.0  			outputfilename = '%s_frame_%04d.png' % ( self.app.Data.getFileLevel('name'), f )
  1667       728      14607.0     20.1      0.1  			outputfilepath = os.path.join( outputpath, outputfilename )
  1668       728   18591853.0  25538.3     79.5  			img.save( outputfilepath, format='PNG' )
  1669                                           
  1670                                           			# keep track of all the processing we've finished
  1671       728       5731.0      7.9      0.0  			processedData[ str(f) ] = outputfilepath
  1672                                           
  1673         1     115963.0 115963.0      0.5  		self.app.Data.setFileLevel( 'processed', processedData )
  1674         1        252.0    252.0      0.0  		self.app.lift()
  1675         1        213.0    213.0      0.0  		self.load()

[mr-martian]

Originally posted by @mr-martian at 2019-10-20T21:10:24Z

Based on this: python-pillow/Pillow#1211

I changed the PNG compression level from the default (6) to 1, resulting in a 10% increase in file size (168.4 MB vs 152.7 MB) and a 57% increase in speed (on the Mac in the lab 37s vs 85s).

[mr-martian]

Originally posted by @jonorthwash at 2019-10-21T05:27:56Z

Ah, right, png compression is slow. This seems like a reasonable trade-off for now.