CHANGES.txt

V3.9.4:
Users:
  - Edit set protocol works with complex sets (titlseries, Classes2d) using "Sub item formula"
  - Adding a prefix to the calculated columns for the metadataviewer
Developers:
   - execution mode moved to class level
   - classifyItems: report detailed information when updateItemCallback fails
V3.9.3: hotfix
Users:
   - Import STOP_STREAMING button is fixed
Developers:
   - stop_import function needed the context for the button action
V3.9.2:
Users:
   - Fix: 'Volume' object has no attribute 'getCompatibilityDict' when using join sets with Volumes 
Developers:
   - SetOfData replaces SetOfStats for generic tabular data.
V3.9.1:
 Users:
   - Fix: CrossSubset protocol works with "id" or "_objId" field too.
   - Fix: Edit set protocol works with complex sets (titlseries, Classes2d) but not editing subelements for now.
 Developers:
   - New constants for "id" and "_objId": ID_COLUMN='id', ID_ATTRIBUTE='_objId'
   - ImageReadersRegistry closer to image handler. Reader (tif, Pil) have write method. Tests added.
   
V3.9.0:
 Users:
   - Chimera viewer compatible with newer versions (flatpak, deb, windows via WSL) via linking

 Developers:
    - Generate a stack of images or a volume from a list of PIL images using ScipionImageReader.

V3.8.0:
 Users:
    - Filehandlers for cs, star and sqlite files to be displayed in the Scipion metadataviewer.
    - Generalizing the method for checking the match between files and objects identifiers
    - New variable: SCIPION_EM_NEW_FILE_CHECK_SEC, default to 10:  defined in the environment determines the number
                    of seconds to wait before checking if new files are available or finished in streamified import
                    protocols.
    - Fix volume halfmaps wrong filename
    - Join heterogeneous sets are possible for tomography objects TS and tomograms.
    - Adapted to new changes in metadata-viewer (v.2.0)
    - xmippLib missing message less critical for those domains without xmipp need.
    - Displaying calculated columns from the transformation matrix
    - Durations like timeouts in streaming can be specified as "1h 24m 36s"

V3.7.1:
    - Hot-fix: SetOf3DClasses and other sets without visualization config can be shown in the metadataviewer

V3.7.0:
  Users:
     - Allowing drag and drop a coordinate when picking
     - Allowing to create subsets from the selection in the metadataviewer plotter
     - Fixing an error creating subset through new metadataviewer
     - NEW PROTOCOL - Crossed subset protocol: Creates a subset of the main set based on a matching field in another set.
     - ChimeraX resolution viewer compatible with WSL.
     - Fix: Wizard for choosing a Mask radius returns integer.
     - Describes better and register variables for the new GUI to edit variables
     - NEW PROTOCOL - data summary: Aggregates (COUNT, MAX, MIN, AVG) any set data based on its fields
     - Importing coordinates from cryoSPARC
     - Fix: The coordinate viewer shows the complete list of micrographs when a subset has been generated and there are only coordinates for some micrographs in this subset
     - Allows to run the whole workflow until 3D processing with trigger points
     - Allowing the use of a pattern for the import of the gain file.

Developers:
     - Plugin._defineVar and _defineEmVar offers  description(str) and  varType(VarTypes) as optional arguments to better describe the variable devel
     - Modify the finishing condition of the parent protocol "extract particles". Now, it checks if the input coords are closed and if all mics and CTFs have been processed.

V3.6.0:
  Users:
     - Adapted to Sprites
     - Add a new 2D coordinate viewer
     - 2 level sets (like tilt series) copy enable flag in subitems when subsetting.

  Developers:
     - Add a new parameter in the classifyItems method(raiseOnNextFailure). This parameter is a boolean flag indicating
       whether to raise an exception if there is a failure while attempting to retrieve the next element from itemDataIterator.

V3.5.1: hotfix tif reader dimensions
V3.5.0
  Users:
     - add wait protocol. This protocol wait N seconds and then ends.
       It is helpful to schedule other protocols using the option "wait for"
     - Fix: .em files are handled by xmipp (dimensions now are correct)
     - Add symmetry option to export_DB protocol. This will produce the list
       of symmetry matrices that PDBe asks for when deposition a new atomic structure
  Developers:
     - Add method setWrongDefocus to CTFModel (code centralization)
     - Add FlexHub model (objects and base methods)

V3.4.0:
   Users:
     - Fix: CTFTomoSeries manual subset should work
     - Windows' ChimeraX properly called from WSL installations
   Developers:
      - add attribute isSubParticles to SetOfParticles
      - modify __str__ to print SubParticles when appropriate
      - remove setOf in SetOfImages.__str__
     - EMSet.copyItems and createCopy accepts "itemSelectedCallback" falling back to isEnabled.
     - getVersionFromPath improved: works with cuda-11.4.safe (uses regex looking up for numbers)

V3.3.1:
     - Improving performance of the metadataviewer.
     - Metadataviewer integration with Chimera and ImageJ     - Subsets: Tolerates missing properties in sqlite but present in run.db (CTFTomoseries without TS pointer)

V3.3.0:
   Users:
     - Fix: importing non cubic volumes could render 0 elements in dimensions. Now minimum is 1.
   Developers:
     - ImageHandler.getDimensions uses ImageReader registry and its readers (Xmipp, Tiff, Eman, ...) image
     - tiff dimensions (gain, eer, tif, tiff) are swap. They original  y were opposite to xmipp and others convention.
     - User subset: ready for new metadata viewer using txt files with ids.
     - classifyItems method of SetOfClasses updates an existing output set than can be utilized for streaming protocols
     - New metadata viewer(BETA). To use it run 'scipion3 pip install metadata-viewer mrcfile'. Right click on outputs should show "Scipion metadata viewer"

V3.2.2; Hotfix: buggy logline commented

V3.2.1: Hotfix: set dimensions are updated upon set creation

V3.2.0:
 Users:
   - New viewer: "Basic MD viewer" to allow doing manual subsets on those sets where before it wasn't possible (Coordinates2D, FSCs)
   - ImageJ filehandler: Set IMAGEJ_BINARY_PATH to the ImageJ or Fiji binary.
   Developers:
     - SetOfCoordinates tweaks: better use of micrographs pointer

V3.1.0
 Users:
   - New Angular distribution viewer (right click on any SetOfParticles or SetOfSubtomograms will list it)
   - Use main color config variable
   - New variable: CHIMERA_OLD_BINARY_PATH: Full path to chimera binary. If defined, it will offer BASIC visualization of pdb and volumes
   - Set editor: summary show the formula
   - Join sets: clones all attributes, not only 1st level ones.
   - New protocol box size checkpoint to validate an estimated particle size.
   - The browser thumbnail show origin info for mrc files
   - Import volumes set the volume comment to the original file.
   - Fix subset: Scheduling waits when subset uses other set
   - ChimeraViewer: Handle also set of volumes and 3d classifications

 Developers:
   - SCIPION_CANCEL_XMIPP_BINDING_WARNING defined in the environment will cancel the warning about xmipp binding missing
   - SCIPION_CANCEL_XMIPP_BINDING_WARNING defined in the github action
   - function readFromPDBDatabase tries to download an atomic structure up to 5 times before returning  an error (before it tried only once)
   - function retry search for fatal error messages in the output and if found,  it aborts instead of trying several times.
   - plotter.py uses logging
   - Plotter.plotAngularDistribution refactored: uses scatter instead of plot fo each point. It may improve performance.
   - Plotter.plotAngularDistributionFromSet new: To plot sets with items.getTransform()
   - Volume.getHalfMaps: receives optionally "asList" to return the list instead of the string
   - Volume.fixMRCVolume, now receives (optionally) a boolean to set the header's sampling rate of the MRC file it refers
   - showj.py: uses logger
   - showj.py: javaVersion cached, javaBin taken from Config.JAVA_HOME o "java"
   - showj.py: SCIPION_JAVA_HOME takes priority over JAVA_HOME (pycharm debugging case)
   - Use new Config.isScipionRunning to register file handlers.
   - Use new Config.isScipionRunning to print xmipp ghost message.
   - SetOfImages._samplingRateStr extracted to convert sampling rate to a str

V3.0.27
   - Hot fix: Assign angles assign full shifts by default. Warns about loosing xFrac and yFrac comming from extract coordinates.

V3.0.26
   - Hot fix: Import volumes annotates the filename in the volume/s

V3.0.25
   - Hot fix: protocols.conf is packaged now!

V3.0.24
users:
   - Version compatible with latest released scipion-em-chimera plugin.
developers:
   - biopython updated to 1.79
   - Using Alphabet from new version of biopython, related tests modified and added.
   - DataViewer allow for registration of new sets using: DataViewer.registerConfig(type, configDict).
     Useful for tomography sets and avoiding unnecessary inheritance.
   - Icon is a png (compatible with future pillow versions)
   - weakImport("tifffile") added when registering FileHandlers (bug at installation in github action)


V3.0.23
users:
   - Subset protocol optimized, specially for small parametric subsets (random or by ids)
   - Assign angles protocol optimized.
   - Filebrowser: thumbnails info shown always without the need of loading viewers.
   - Thumbnails for mrc files (and its flavours) show also headers sampling rate.

developers:
   - Plugins discovery takes into account priority packages: pwem, tomo, ..
   - FileHandler registration moved to main pwem.__init__.
   - Import volumes uses logging
   - Import volumes fixed when downloading an EMDB.
   - Dataviewer allow registration of other sets. Dataviewer: DataViewer.registerConfig(tomo.objects.SetOfTiltSeriesCoordinates)

V3.0.22 (hotfix)
   - Plugin.guessCudaVersion returns Version("0.0") for cases like this /usr/lib/x86_64-linux-gnu

V3.0.21
users:
   - New protocol: Mathematical operator.
   - Import sequence protocol can now download sequences from GeneBank. In the past only 
     UniProt was accessible. In general Unitprot does not have information about 
     proteins not well know AKA  "uncharacterized proteins" 
   - "import particles" protocol accepts cryoSCPARC particles files
   - Import coordinates fixed for complex streaming cases.
   - Extract particles does not need SetOfMicrographs is "same as coordinates" is used.
   - AtomStruct attributes viewer now can plot attributes over the structure sequence

developers:
   - Methods to guess the cuda version from a variable that point to a folder in Plugin class:
     (Plugin.guessCudaVersion(varname)). Follows links.
   - Allow indirect pointers in SetOfClasses.
   - Warning added in SetOfCoordinates.setMicrographs when not using an indirect pointer.
   - Warning added in SetOfClasses.setImages when not using an indirect pointer.
   - data.py test_workflow_mixed.py and uses logger instead of print

V3.0.20
users:
   - fixes numpy dependency conflict PyWavelets for python3.6-> scipy <= 1.8.3

V3.0.19
users:
   - fixes numpy dependency conflict PyWavelets -> scipy 1.9 --> numpy > pyworkflow numpy.

V3.0.18
users:
   - "assign ctf" protocol now accepts SetOfParticles too.
   -  "subset" protocol accepts now a range of ids.
   - FIX:import atomic structures now works also when files are named the same but in different folders.
developers:
   - New model class: SetOfPrincipalComponents(SetOfNormalModes)
   - New base wizard: VariableWizard - Wizard base where input and output paramNames can be modified and added, so
     one wizard can be used in several protocols with parameters of different names.
   - SelectChainWizard replaces GetStructureChainsWizard and uses VariableWizard
   - SelectResidueWizard added: To select a residue
   - SelectAttributeWizard added: Wizard to select attributes stored in a scipion object or set

V3.0.17
users:
   - Updating protocol box size related parameters. Now for sanity reasons, it outputs only even numbers.

V3.0.16
users:
   - Fixing an error calculating the FSC resolution value. 
   - FSC plots legend shows "resolution" ;-)
   - Align movies protocols: possible outputs for even odd mics defined
   - Import volume: fix downloading some old EMD id not numeric
   - Hdf files can be open from the file browser
   - Adding the protocol "box size related parameters" to estimate the different parameters needed for various pickers.
      
developers:
   - New method at pwem/convert/headers.setMRCSamplingRate: sets the sampling rate of an mrc file. Image Object has it: volume.setMRCSamplingRate(3.5) should work.
   - protcols.conf minimized only containig its own protocols
   - Variable wizard with customizable target, input and output parameters


V3.0.15
users:
  - New viewer: ChimeraAttributeViewer: analyze scalar attributes of the residues or atoms in a structure
  - Local resolution viewers use now native ChimeraX key
  - Adding the protocol "Manual check point" that kept running for a time determined by a parameter or until
    the user determines it is convenient.
  - Calculating the FSC resolution value and displaying it in the viewer.
V3.0.14
users:
  - New protocol: numeric classes extractor: Extracts items from a SetOfClasses based on number of items assigned to the classes
  - Filter ser protocol can filter by ranking: ascending or descending getting N items or a percentage.
  - Outputs predefined for:
     import masks
     import mics
     import movies
     movie alignments
     ctf estimators
     pickings
     extraction
  - Filebrowser preview for images limited to 500MB (default). MAX_PREVIEW_FILE_SIZE (in MB) can be used to change this value.

V3.0.13
users:
  JAVA_MAX_MEMORY defaults to 4GB for a better user experience with large datasets
  Take CHIMERA_HOME from chimera plugin if available. Otherwise will go for chimerax-1.2.5 as default
  Clean problematic characters in filenames: ;<>?"()|*\'&
  "filter set" protocols extended to filter by coordinate distance.
  "edit set" protocol extended to allow editing angular information.
  .gain files are treated as tiff files when getting its dimensions.

developers:
  Transform class provides a "create" (factory method) to create common transformations.
  pwem.convert.headers.fixVolume: to fix mrc headers of volumes.

V3.0.12
users:
  fix create user subsets with classes2D

developers:
  pwem.convert.headers.fixVolume: fixes the header of a volume defined as stack, so we can avoid :mrc suffix in paths the paths.
  images imported by ProtImportImage and ProtImportMask remove problematic characters -> PROBLEMATIC_SHELL_CHARS = ;<>?"()|*\'&

V3.0.11
users:
  mrc, pdb, vol files can be open with Chimera from the file browser
  rec file can be imported (not yet displayed by xmipp)
  fix import of particles stack with one image

developers:
  EmSet has a create copy method to properly copy a Set
  useful for generic set handling (subset, join set, user subset)
  tilt series subsets bug fixed
  MaskVolumeWizard - Creates a mask for a volume interactively. Masks currently implemented: Spherical mask

V3.0.10
users:
  "set filter" protocol and wizard
  protocol tree (left panel) collapsed at loading
developers:
  protocols.conf created taken from scipion-app defining its own protocols.
  Add getTransformMatrix function, finds transformation matrix between to PDB files
  Add partial support for eer files, just reading dimensions (part 1)