v1.7.2 - RollupCalcItemReplacer bugfix

• Fixes issue reported with RollupCalcItemReplacer incorrectly detecting Type__r fields as polymorphic

v1.7.1 - Scheduled Rollup Updates & Date literal comparisons

• Fixes #640 by properly ensuring RollupDateLiteral comparisons occur in local time since they are always based off of the running user and their time zone
• Fixes an issue brought up in #639 where scheduling a rollup can inadvertently exceed the max query rows

v1.7.0 - Improve Memory Management for LDV Full Recalculations

This is the first (potentially) non-breaking change medium version bump, but in reflecting on the scope of the changes being made, it feels appropriate to make the leap to version 1.7.0. Here's a breakdown of the changes:

Overview

• RollupFullBatchRecalculator is no longer an actual batch class, per se, though it still does batch processing. Instead, it takes advantage of the Apex Cursor beta to iterate through children records when doing full recalculations where the number of records exceeds the value for RollupControl__mdt.MaxLookupRowsBeforeBatching__c. Like its batch class predecessor, it will continue to default to calculating rollup values for 500 children at a time - still configurable using RollupControl__mdt.BatchChunkSize__c. Using Database.Cursor has several big advantages over using a batch class, and while the feature is still in beta, I expect that it will GA soon
• Previously, Apex Rollup took advantage of Database.Stateful, which is a marker interface that's part of the Database.Batchable framework to calculate what a parent record's prior value was when that parent record had children split between batches. This happens more often than you'd expect, even with very fine-tuned ordering of the children records. With large data volume Salesforce orgs, keeping track of which parents have been updated while iterating through all of the children records represents a serious problem, and at a certain point when relying on Database.Stateful to calculate millions of records and dozens of rollups for a single child type, the rollup calculation will inevitably fail as memory runs out.
• While managing everything in-memory is extremely convenient as the package author, I'd basically arrived at the end of the road when it came to optimizing the batch full recalc process to try to avoid running out of memory. There were no tricks left - just the cold, hard fact that 12 MB of space will eventually run out with a one-to-many relationship between each parent record and the rollups being applied to that record when there are millions of parent records involved. This was the problem reported in #615
• ⚠️ Salesforce orgs have a MAXIMUM number of active DataWeave scripts that can be used. ⚠️
  If you saw the phrase "(potentially) non-breaking change" above and went 🤔, v1.7.0 adds a DataWeave script to the package. While I view it as unlikely that there are orgs out there running up against that limit, it's worth calling out here. Contact your AE if you truly cannot accommodate additional active DataWeave scripts and want to update. See the section titled "Easily Parsing Data" below for more info on why this change is being made
• Because records are now created during "batch" full recalculations, if you abort a full recalculation using the Apex Jobs page in setup, or via some other means, there is the potential that RollupState__c records will be orphaned and won't be cleaned up. You will have to delete those records manually. More info is available in the "Easily Deleting Data" section.

Challenges

There's one natural conclusion when an in-memory data management strategy is no longer sufficient, and that's a record-based memory management solution. As a package author, however, it would be irresponsible (not to mention untidy) to implement a feature that contributes significantly to the data storage on a subscriber org. This means that the "easy" approach - a data record (henceforth referred to as "state", or "Rollup State", or the API name for the object) for each parent - was immediately disqualified. The solution would have to compress data such that the previously calculated values for a parent record could be:

• easily retrieved,
• easily parsed
• easily deleted

Easily Retrieving Data

In order to facilitate the easy retrieval of data, v1.7.0 uses a new custom object, RollupState__c, which has on it 11 different custom text fields ranging from RollupState__c.RelatedRecordKeys0__c all the way through RollupState__c.RelatedRecordKeys10__c. Because you can't filter on long text area fields within SOQL, and because the maximum length of a standard text field is 255 characters, creating many fields like this is a crucial piece of the data compression strategy that makes it possible to store state for many parents within a single record. Assuming that most rollups will be keyed to standard Salesforce ids, it follows then that each related record key field can hold (255/(18 + 1) = 13 parent ids per field (the + 1 accounts for the comma separating each key).

Despite using 11 custom fields for this (which might sound like a lot), that ends up allowing for a maximum of 13 x 11 = 143 parent records' data being compressed into a single RollupState__c. Now that the kinks have been ironed out with the logic that keeps track of which related record key field is being used at any given time, it will be possible to further expand the number of custom fields being used if there's any concern with the data storage use of Rollup State records for these calculations (which would allow for the data to be even further compressed). That being said, since relevant records are deleted within each cursor "chunk" that's processed, I am hopeful that the data storage concerns being discussed here are purely theoretical.

Easily Parsing Data

This proved to be an unexpected hurdle - initial benchmarking on deserializing the long text area on each Rollup State record was poor. As in, for some chunks that had only a dozen state records, the deserialization of those records back into Apex-defined types was causing the 60 second Apex CPU time limit to be exceeded 🤯.

This necessitated the next big change to Apex Rollup - the introduction of a DataWeave script to perform the deserialization. Using DataWeave, the deserialization time was cut from a process that at times was taking 30+ seconds to an operation that has always finished within a single second. That's enough of a justification for me to proceed with this piece of tech, though I do still have plans to circle back on the poor performance of the native deserializer.

Easily Deleting Data

At the end of each recalculation job, any remaining RollupState__c records are deleted. This is something to be mindful of. If you were to abort a currently running recalculation job, this would effectively "orphan" those records, and they'll need to be deleted separately (by you, presumably). The framework will not (at this time) do any periodic maintenance on removing older RollupState__c records that could be orphaned in this manner, though I am open to the idea of giving users a way to conveniently do so via the Apex Rollup app should the need for that arise.

Special Considerations

One of the... downstream effects... of moving away from using Batch Apex was that the majority of the Apex Rollup framework now relies on Queueables to perform async processing. In order to accommodate this, v1.7.0 takes (greater) advantage of Queueable Finalizers to coordinate work between different threads, as finalizers are the only place where it's possible to fully chain different jobs together without exceeding limits.

As I had many failing tests when replacing the Batchable version of RollupFullBatchRecalculator with a Queueable, and I've been performing integration testing to sanity check huge swathes of this work, I don't expect that this change should be visible to people in anything other than a possible increase in Debug Logs being generated (as Finalizers create their own, separate, log file). I've mentioned it here, though, on the off-chance that this change does somehow lead to deleterious effects.

---

A huge thanks to @jongpie for reviewing #636 for this, and for being an invaluable colleague and friend

v1.6.36 - Patches RollupFlowBulkProcessor + Rollup Order By support

• Patches RollupOrderBy__mdt issues when called from RollupFlowBulkProcessor (specifically for multiple order bys)
• Updates API version for metadata from 61.0 -> 62.0

v1.6.35 - Rollup Order By Bugfix

• Fixes an issue reported where rollup ordering wasn't being properly applied for rollups with multiple order bys
• Fixes #629 by properly handling text template HTML tags (when present) for the Full Recalc CMDT-driven Invocable rollup action

v1.6.34 - Bugfixes!

• Fixes #619 by more carefully handling parentRecordIdForEmptyChildrenCollections variable
• Removes heap size checks as they are unncessarily expensive when checking rollup limits
• Fixes #626 by patching a few places where RollupControl__mdt being null (which is sometimes set internally by the framework for performance reasons) does not cause issues when trying to log
• Fixes #623 by properly parsing nested IN conditions during recursive where clause operations
• Fixes #622 by properly tracking changes to calc items when updates occur with differing values for multicurrency orgs
• Fixes an issue where sync rollups enqueued by Flow would not run in the proper order
• Attempted fix for #625 - do not allow max query rows to exceed the platform limit

v1.6.33 - THIS_MONTH date literal bugfix

• Addresses issue reported on SFXD with THIS_MONTH date literal in APAC locales
• Investigating #616 with some full batch recalculation housekeeping

v1.6.31 & v1.6.32 -

Two Releases In One!

1.6.31

• Updates pipeline to only run SF scanner for dependabot PRs
• Fixes issue discussed in #562 where sync rollups using lowercase relationship queries in where clauses would not start up properly

1.6.32

• Adds overload for Rollup.runFromApex(List children, Map<Id, SObject> oldChildrenMap, System.TriggerOperation rollupContext) thanks to @arbokrad for the feedback!

v1.6.30 - Updates Extra Code Coverage Plugin

• Fixes #607 by updating Extra Code Coverage plugin. Sews up the behavior of RollupFlowTests.shouldReportErrorWhenConcatDelimiterSetAndNotConcat - thanks to @bfalter for reporting
• Updates sfdx-scanner dependency
• Streamlines a few more ternaries into null coalesces
• Takes advantage of the new defaultValue invocable variable properties to give Flow builders visible guidance when appropriate

v1.6.29 - Grandparent Full Recalc Bugfixes

• Fixed an issue with grandparent rollups where all parents couldn't be retrieved in one go due to polymorphic fields being used
• Fixes an issue reported by aesthetics where grandparent full recalc rollups with mutually exclusive where clauses could accidentally cache the previously filtered children from prior where clauses