0.0.30 release

Synapse v0.0.30 Release Notes

New Features

• #473 - Added it:prod:soft, it:prod:softver, it:hostsoft types and associated forms, to allow tracking software, versions of software and software installed on a given host.
• #473 - Added it:semver data type for doing type normalization of Semantic Version numbers. Added helper functions for both Semantic version parsing and generic version parsing to synapse.lib.version
• #473 - Added it:version:brute typecast to attempt parsing a version string into a normalized system value that can be used to do ordered comparison of version strings.
• #476 - Added inet:iface type and form for modeling a network interface on a particular device being bound to a particular IP, host, phone or wifi SSID. Added inet:wifi:ssid type.
• #476 - Added ps:contact type and form to act as a conglomerate of contact information for a individual.
• #476 - Added tel:mob:tac, tel:mod:imei and tel:mob:imsi types and forms for modeling cellphone related information. This includes parsing and validation of pre-2004 IMEI/IMSI numbers.
• #482 - Moved test helper functions from synapse.tests.common to synapse.lib.iq so other users of Synapse can reuse our pre-existing test helpers (SynTest, TestEnv and TstOutput).

Enhancements

• #465 - Added axon:listener, axon:tags and axon:syncopts to the AxonHost class, so these default values can be passed to Axons made by an AxonHost.
• #479 - Add test for calling the storm task() operator on a remote cortex with a local calback handler.

Bugs

• #475 - Changed PropValuType to use reqPropNorm instead of getPropNorm to enforce that the property referred to BY the type must be a modeled property.

0.0.29 release

Synapse v0.0.29 Release Notes

New Features

• #471 - The dmon tool, synapse.tools.dmon can now accept the log level via a environmental variable, SYN_DMON_LOG_LEVEL. This can be added as an environmental variable in a docker compose file using the Vertex Project Synapse Docker images to configure the logging level.

Enhancements

• #467 - Added it:exec:proc:path, it:exec:proc:src:proc and it:exec:proc:src:exe properties to the it:exec:proc form. Removed the it:exec:subproc form since it is not needed with the :src: properties on it:exec:proc.
• #467 - Removed the computer science model (compsci.py) since it was superseded by the host execution model.
• #467 - Added inet:flow:src:exe and inet:flow:dst:exe properties to inet:flow to allow modeling data between file:bytes nodes.
• #468, #469, #472 - Added pytest-xdist to the testrunner.sh script to speed up local (dev) test runs of synapse.
• #470 - Remove unnecessary docker related functionality.

Bugs

• #466, #474 - Fixed bug in non-blocking SSL link which would sometimes prevent data from being transmitted

Documentation

• #462 - Added documentation for the host execution model in infotech.py.

0.0.28 release

Synapse v0.0.28 Release Notes

New Features

• #456 - A global thread pool has been added to Synapse and a Task object convention added for executing tasks in the pool. This is in preparation of future feature support.

Enhancements

• #461 - The storm query operator refs() now also lifts nodes by prop-valu combination in order to get nodes which may refer to the source nodes. This allows identifying XREF nodes which point TO the inbound node.
• #463 - The TimeType now norms the string "now" as the current system time.
• #464 - Added a "guid" helper for Ingest to assist in making GuidType nodes without having to form strings out of variables.

0.0.27 release

Synapse v0.0.27 Release Notes

New Features

• #446, #450 - Adds the ability for GuidType nodes to normalize a list of items, in order to generate stable guids for potentially re-encounterable data. This only works when generating a property norm value (getPropNorm) and does not work for purely type normalization (getTypeNorm). Storm keyword list argument parsing can be used to generate stable GUID using the CLI, Ingest or Storm mechanisms.
• #452 - Synapse now stores the current version of the Synapse library in the Cortex blob store at the end of Cortex initialization. This was done in order to prepare for eventually enforcing required upgrade paths for data migrations or other features.
• #447 - Added the new form inet:web:postref XREF to track an inet:web:post which refers to another node.
• #447 - Added the new form inet:web:action GUID to to track an arbitrary action by an ``inet:web:acct`. The actions tracked by this are by defined by [Synapse] user convention.
• #447 - Added the new form inet:web:actref XREF to track how an inet:web:action may have interacted with another node.
• #454 - Added the inet:web:chprofile GUID to track previous values of a inet:web:acct node, representing changes to user accounts or profiles.
• #454 - Added the inet:web:post:repost property to track the concept of a inet:web:post being a copy of another post.
• #455 - Added a pair of Storm (and Cortex) Configable options to enable and set logging levels for Storm queries. These are storm:query:log:en and storm:query:log:level. This logs what the query is and what the user execution context was.
• #426 - Axon and AxonHost objects are now Configable objects with configuration definitions that are used to define their behavior.

Enhancements

• #442 - Python 2.7 support dropped from Synapse.
• #447 - Migrated all inet:net* forms to the inet:web:* space. The following is a map of the migrated forms and their corresponding new forms:

('inet:netuser', 'inet:web:acct')
('inet:netgroup', 'inet:web:group')
('inet:netmemb', 'inet:web:memb')
('inet:follows', 'inet:web:follows')
('inet:netpost', 'inet:web:post')
('inet:netfile', 'inet:web:file')
('ps:hasnetuser', 'ps:haswebacct')
('ou:hasnetuser', 'ou:haswebacct')

These forms will automatically be migrated in existing Cortexes. If XREF types were used to point to any of these forms and the cortex was not first migrated to v0.0.26, the XREF type migration will fail. It is recommended that users first upgrade to v0.0.26 prior to upgrading to v0.0.27.

• #447 - Added Storage.updateProperty() and Storage.updatePropertyValu() APIs to the Cortex storage layer for doing bulk property and property-by-value updates. These are explicitly NOT exposed in the Cortex class.
• #449 - Thinned out some components of the EventBus class for performance reasons. This did result in the removal of the synapse.eventbus.on() decorator for decorating functions to be used as event callbacks.
• #456 - Removed unused Synapse modules: synapse.hivemind, synapse.mindmeld, synapse.lib.moddef.
• #426 - Logging in tests is now controlled by the environmental variable SYN_TEST_LOG_LEVEL which, as an integer, will set the logging level used by the root logger.
• #426 - The environmental variable SYN_TEST_SKIP_LONG can be set to a non-zero integer to skip potentially long running tests. This can shave up to a minute of test execution time.
• #426 - Axons now have the in-memory cache enabled on their Cortex by default.
• #458 - The inet:web:acct:occupation property has been changed from a str:txt type to str:lwr to allow for better foldability between user-declared occupations.

Bugs

• #443 - Make the daemon return more useful error messages when an exception has occurred during execution of a remote request.
• #444, #445 - Allow an inet:srv4 type to accept an integer string as input. Also adds additional boundary checking when norming an ip:port string to ensure that irreversible inputs are not accepted.
• #453 - Cleaned up skifIfNoInternet() test helpers. They will now be allowed to fail unless the SYN_TEST_SKIP_INTERNET environmental variable is set to a non-zero integer.
• #426 - The synapse.lib.heap.Heap class was not properly responding to heap:resize events. This was remedied.
• #426 - Wrapped a .items() iterator in synapse.daemon.OnHelp with a list to prevent a RuntimeError.
• #426 - Fix the synapse.lib.service.SvcProxy.getSynSvcs() method to return a Telepath safe list instead of a dict.values() view object.
• #426 - Fix the synapse.lib.service.SvcProxy.init to strap in event handlers AFTER initializing instance variables to avoid a race condition on startup.
• #426 - AxonHost now waits before advertising itself on the bus, and properly calculates the number of axons it needs to make. This addressed an issue where the AxonHost was generating a non-deterministic number of Axons.
• #426 - The Axon’s thread to make clones for itself on a ServiceBus now waits until a remote clone is made. This addressed an issue where the Axon would make extra clones for itself.

Documentation

• #448 - Added in-model documentation for the file: model defined in files.py.
• #451 - Added user guide information for Storm lift operations, lift(), guid() and alltags().
• #426 - Docstrings in the synapse.axon module were rewritten or added when needed.

0.0.26 release

Synapse v0.0.26 Release Notes

New Features

• #438 - Added PropValu datatype to synapse. This allows a secondary (or primary) property to be modeled as a string in the form property=<repr valu>. This type also yields sub of "prop" representing the property, and "strval" or "intval" being the system normalized value of the property. This allow for node creation where a reference to another node property is needed but cannot be defined up front in the model. The additional subs allow for filtering/pivoting operations on nodes which use the PropValu type. A simple example of the string form for a PropValue is inet:ipv4=1.2.3.4. A more complex example of the string form for a PropValu is inet:passwd=oh=my=graph!
• #438 - XREF types have been updated to use the PropValu datatype instead of storing data in the property columns. This removes any extra-model data from the property columns in Cortex rows, and allows the implementation of pivotable XREF nodes. The string syntax for XREF creation was changed to be in line with the Comp datatype syntax, which looks now looks like ``. This will affect any ingests or programmatic creation of xref nodes done by users. An example of the string form of a Xref now looks like the following: (98db59098e385f0bfdec8a6a0a6118b3,"inet:passwd=oh=my=graph!"). Note that the PropValue portion of the value is quote delimited.

Enhancements

• #438 - The refs() storm operator is updated to be aware of secondary properties which are PropValu types and will pivot off of them, even if the props themselves are not forms.
• #438 - The file:imgof and file:txtref nodes no longer have glob property xref:*, instead :xref is now a PropValu and has the additional :xref:prop, :xref:intval and :xref:strval secondary props. This does require a data migration, so deployment of v0.0.26 should be tested on Cortexes which use those forms prior to production use.
• #438 - Migrated unittests to using getRamCore() helper when possible.
• #438 - Added unittest self.len() helper. Started some migration there.
• #439 - Cleaned up Socket() class implementation to be better, stronger and faster.
• #440 - Moved the syn:type, syn:form and syn:prop forms (and other items declared in DataModel) to be part of the "syn" model. These core elements are now themselves introspect-able.
• #440 - Added syn:prop:base and syn:prop:relname to syn:prop nodes, so those nodes can now be lifted by a basename or a relative property name.
• #441 - Added Python 3.7 RC1 to the test matrix. Pinned postgres test image to postgres:9.6.
• The Storm runtime now respects the storm:limit:lift configable value as the default limit when lifting nodes.

Bugs

• #440 - Changed how runtime nodes (runts) used to represent the data model in a cortex are created. These are now made from the type and property definitions based on the loaded data model; having been guided by the data model being processed. This restores things like "syn:prop:form" which was accidentally dropped from nodes.

0.0.25 release

Synapse v0.0.25 Release Notes

New Features

• #404 - Added model for inet:web:netlogon to track netuser's logging into services.
• #433 - Added inet:dns:mx, inet:dns:cname, inet:dns:soa, inet:dns:txt node types for recording different DNS responses.
• #436 - Added the ability to define triggers. Triggers are predefined actions which can react to events and trigger storm queries, in order to automate activities and actions.
• #436 - Rewrote the user authentication system to be more integrated with Cortex (and used within the trigger subsystem). This allows adding user roles and permissions to add, delete, or update nodes.

Enhancements

• #404 - Added inet:whois:nsrec comp nodes to track nameserver's associated with whois record. This removed the inet:whois:rec:ns* properties and migrates existing props into the new nodes.

Bugs

• #437 - Fix the tag interval filtering in the Storm query system.

Documentation

• #435 - Rewrote docstrings for synapse.lib.queue subsystem.

0.0.24 release

Synapse v0.0.24 Release Notes

Enhancements

• #430 - CortexTest class was split into CortexBaseTest and CortexTest. CortexBaseTest is used for running the
  basic_core_expectations tests with the different storage backing; and CortextTest is used for more generic API tests.
  This allows running the basic test suite against the different storage types without running the entire test suite, in
  order to isolate possible storage related issues.
• #434 - Migrated from using nosetestes to using py.test as the Synapse testrunner for CI.

Bugs

• #422 - Fixed the order of autoadds being added to the DB during the addTufoEvents / addTufoEvent Cortex API. This
  could manifest itself as tufos made by these events to not contain their properties.
• #428 - Fix a issue with the LMDB cortex where parameters passed to pylmdb were not bool types (as that library
  expected), causing the parameters to be ignored in effect.

Documentation

• #422 - The inet:dns model docstrings were updated to be more comprehensive.

0.0.23 release

Synapse v0.0.23 Release Notes

New Features

• #423 - Added ephemeral runtime-only nodes as a concept to Synapse. The data model has been migrated to utilize these for loading and storing the data model, so the data model (syn:type, syn:prop, syn:form) itself no longer lives within the Cortex database itself. This makes data model updates much easier, requiring only data migration functions to be written for future updates. This does mean that custom models are no longer persistent and are required to be loaded into a Cortex to be made available.

• #423 - Added a initial model for capturing software execution knowledge on hosts. These models are primarily comp types, which allow modeling varying levels of knowledge which may be available from different data sources.

Enhancements

• #423 - The setModlVers API was moved to the Storage layer (and simply called through by the Cortex) to allow hooking model revision function execution with events, to allow for easier testing of Model data changes requiring data migrations.
• #429 - Updated .drone.yml to run all tests in parallel again to take advantage of infrastructure updates.

Bugs

• #421 - Fixed getConfDefs() API in Configable. Made it so that default values (defval) items are copy.deepcopy'd, so that mutable defval's are not overwritten by later use.
• #425 - Prevent the Ingest tool from attempting to ingest data to a remote Cortex connected to over Telpath. Attempting to do so previously would result in a esoteric error message. This means that the --sync option must be used when syncing data up to a remote Cortex.

Documentation

• #424 - Docstrings for the synaspe.lib.config.Config class were written to clarify that class APIs and add notes about possible race conditions when using onSetConfOpt handlers to respond to configable events.

0.0.22 release

Synapse v0.0.22 Release Notes

New Features

• #395 - Add formTufosByProps() API to the Cortex() class to do bulk tufo creation. This has significant performance for doing bulk tufo creation by utilizing a single storage layer transaction.
• #359, #405 - All splices now represent atomic actions. This is a breaking change from v0.0.21, since the node:set events no longer multiple props, but instead a single prop per event. This requires an ecosystem wide upgrade for users utilizing the splice subsystem.
• #408 - Provisional task() operator added to the storm runtime. This fires events in the form of "task:" on the storm core eventbus. This behavior may change in the future.
• #409, #414, #416 - Synapse model properties which had req=1 set on them now actually require that property to be present when model enforcement is in place. This check occurs after node:form events, allowing code to hook the node formation process and provide required properties or set them as needed.
• #417 - Added a tree() operator to storm for doing recursive pivot operations on a set of nodes.
• #419 - Added a delprop() operator to storm to remove properties from nodes.

Enhancements

• #410 - pivot() operator syntax updated to match that of the macro syntax operator. This may be a breaking change for any programmatic use of the pivot operator syntax.
• #411 - Added pycodestyle checks to CI builds to identify code style issues.
• #412 - Fix resourcewarnings with unclosed file handles.

Bugs

• #399 - Fix Cortex.init() on Python 2.7 where a list comprehension smashed function locals. This prevented configable options from being set at initialization.
• #401 - Fix setTufoProps() to add nodes created from secondary properties when appropriate.

Documentation

• #403 - Initial docs for Ingest subsystem,
• #408, #420 - Updates to Storm runtime documentation.
• #418 - Added markdown template to release documentation.

0.0.21 release

Synapse 0.0.21 Release Notes

New Features

N/A

Enhancements

• #299, #381, #392 - Update model enforcement to prevent nodes from being created which are not valid types.
• #391 - Add bumpversion support for doing release cutting.
• #394 - Add log messages when doing model revision migration.

Bugs

• #396 - Fix a bug with Configable objects with telepath proxy attributes.
• #397 - Add a model revision to fix inet:urlfile comp type nodes to address a issue from #333.

Documentation

• #391 - Start release process documentation
• #393 - Additional user guides