Support of Sprig (filter for variables)

[chermed]

Summary

why not integrate a library like sprig (http://masterminds.github.io/sprig/) to be able to transform data without adding extra tasks, example:

          - name: FD
            dependencies: [FB-SPLIT]
            template: curl
            arguments:
              parameters:
                [
                  { name: endpoint, value: "func-external.default" },
                  { name: msg, value: '{"value": {{item}} }' },
                ]
            withParam: "{{tasks.FB-SPLIT.outputs.result | toObject | field .results | toList}}"

What change needs making?

Use Cases

When would you use this?
Let's assume that I have many functions (top of serverless framework) that accepts JSON as input and outputs
The syntax of the input json is {"value": {"itemX","valueY}}
The syntax of the output json is {"results": [{"item0","value0"}, ..., {"itemN","valueN}]}

When building the dag, we need to loop on results and then construct the new JSON. the problem is that we need to extra tasks to make transformation.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

This is a good idea. Would you like to submit a PR? @simster7 might be willing to give pointers.

[alexec]

@jessesuen this go a lot of thumbs-up very quickly - I think it supports your arguments for better in-template data processing features

[alexec]

Our main evaluation library is https://github.com/antonmedv/expr

[alexec]

We use fasttemplate for templates, it is fast, but basic, but it does not Sprig.

There is a lot of code around templating, and I don't think I'd be comfortable changing that code. Instead, I think we can introduce a new field:

 withParamExpression: "map(tasks['FB-SPLIT'].outputs.result, {.results})}}"

[alexec]

This could be solved by plugins obviously. #5025

[max-sixty]

Ref #1293

[simster7]

We're working on implementing this or something similar in #4958. The current spec supports in-memory transformations as well as delegation to a pod if necessary

[max-sixty]

@simster7 tell me if this is chirping from the peanut gallery — but will this approach involve writing & maintaining code within Argo for each transformation? Is there an approach for inheriting functionality immediately from a library like gomplate or sprig, even if it involves another pass on top of fasttemplate?

[alexec]

Question for @chermed @max-sixty, the example in the is withParam. I initially thought you needed more powerful templates everywhere, but is this almost completely a problem for withParam?

[max-sixty]

My original use case was date manipulation, not in withParam, but obv don't overfit on me...

[alexec]

@max-sixty could you provide other examples? withParam is a good example of where using filter and map operations would be powerful, however I'd love to see more use cases to we can more fully understand this.

[max-sixty]

The case we had was having a legible date format for workflow creation time, which ends up being verbose and inflexile:

        - name: run_date
          value: "{{workflow.creationTimestamp.Y}}{{workflow.creationTimestamp.m}}{{workflow.creationTimestamp.d}}"

This would be much easier with sprig / gomplate / etc:

"{{workflow.creationTimestamp | date "2006-01-02"}}"

[alexec]

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: expression-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: task-0
            template: pod-0
            arguments:
              parameters:
                - name: foo
                  value: "{{=int(item) + 1}}"
            # stringify: withParam must be a JSON list encoded as a string
            withParam: "{{=stringify(filter([1, 3, 7], {# > 1}))}}"
    - name: pod-0
      inputs:
        parameters:
          - name: foo
      container:
        image: argoproj/argosay:v2
        args: [ echo, "hello {{=int(inputs.parameters.foo) * 10}}" ]

[simster7]

I think a good distinction between #5115 and #4958 is the scope of the data. Enhanced templating targets quick, in-line, modifications to parameters while the data: template deals with sourcing and transforming a large amount of data.

I like to think of the data: template as bash pipes: trying to replicate an experience such as cat file.txt | sed ... | grep ... | sort | uniq | xargs ...

Enhanced templating would be like doing in-place calls echo $(( $var + 1)) or cat $(pwd)/file.go

[alexec]

I think the scope of this has to be limited. What I can't currently fathom is, expr is missing group and reduce operators. I'm asking them to add them (because adding them myself would require some solid 1st-year CompSci compiler writing skills I seem to no longer possess). I'd like to allow changing the templating library.

[simster7]

What I can't currently fathom is, expr is missing group and reduce operators.

Agreed. This is why I think expr is very limited for this use case

[alexec]

It's more powerful than text/template, but I think we might want something more powerful still. Maybe a mini-scripting language.

[alexec]

See #1293

[alexec]

Example of the proposed solution - PLEASE COMMENT!

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: expression-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: task-0
            template: pod-0
            arguments:
              parameters:
                - name: foo
                  value: "{{=int(item) + 1}}"
            # stringify: withParam must be a JSON list encoded as a string
            withParam: "{{=toJson(filter([1, 3, 7], {# > 1}))}}"
    - name: pod-0
      inputs:
        parameters:
          - name: foo
      container:
        image: argoproj/argosay:v2
        args: [ echo, "hello {{=int(inputs.parameters.foo) * 10}}" ]

[max-sixty]

Thanks @alexec , this looks exciting.

Where is the filter function from? I sees toJson but not filter in sprig? Is it implemented in argo?

[alexec]

filter is expr syntax.

https://github.com/antonmedv/expr

[max-sixty]

Thanks! I see now. But the toJson is sprig? How do those interact?

Overall from the user POV this looks really great — maybe that's the sort of comment you were asking for.