Bug: GET /v1/pipelines should include associated processors

[raulb]

Bug description

Important

Fixing this issue would be nice since we could include the processors as part of conduit pipeline ls before we release 0.13.0

Working on #2019, I noticed that even though a pipeline contains a processor, the PipelineService doesn't include this information, appearing to be empty:

curl -X 'GET' \
  'http://localhost:8080/v1/pipelines' \
  -H 'accept: application/json'  
[
  {
    "id": "add-department",
    "state": {
      "status": "STATUS_RUNNING",
      "error": ""
    },
    "config": {
      "name": "add-department",
      "description": "An example pipeline which reads data (imaginary employees) from two generator sources, processes it and writes it to a file.\nIt attaches the built-in `field.set` processor to one of the sources to add a `department` field to its records. The records from the other source are not processed.\n"
    },
    "connectorIds": [
      "add-department:employees-1",
      "add-department:employees-2",
      "add-department:file-destination"
    ],
    "processorIds": [],
    "createdAt": "2025-01-09T12:23:13.737117Z",
    "updatedAt": "2025-01-09T12:23:13.737693Z"
  }
]

Here's my yaml

version: 2.2
pipelines:
  - id: add-department
    status: running
    description: >
      An example pipeline which reads data (imaginary employees) from two generator
      sources, processes it and writes it to a file.
      
      It attaches the built-in `field.set` processor to one of the sources
      to add a `department` field to its records. The records from the other source
      are not processed.
    connectors:
      - id: employees-1
        type: source
        plugin: builtin:generator
        settings:
          format.type: "structured"
          format.options.id: int
          format.options.name: string
          format.options.company: string
          format.options.trial: bool
          recordCount: "1"
        processors:
          - id: extract-name
            plugin: field.set
            settings:
              field: '.Payload.After.department'
              value: 'finance'
      - id: employees-2
        type: source
        plugin: builtin:generator
        settings:
          # department collection
          format.type: "structured"
          format.options.id: int
          format.options.name: string
          format.options.company: string
          format.options.trial: bool
          recordCount: "2"
      - id: file-destination
        type: destination
        plugin: builtin:file
        settings:
          path: ./example.out

Steps to reproduce

1. Make sure you have a pipeline with a processor.
2. Run conduit
3. Visit http://localhost:8080/openapi/#/PipelineService/PipelineService_ListPipelines and inspect the response on this request.
4. Notice processors are not included on the pipeline that has it.

Version

v0.13.0-nightly.20250110

[hariso]

@raulb Now I noticed that the processor extract-name you have is attached to the employees-1 source connector. That's why, when you get your pipeline through the API you don't see the processor, because it's not attached to the pipeline itself. If you fetch your source connector through the API or list all the connectors, you should see the processor.

In other words, when you fetch a pipeline(s), the response is meant to include just the pipeline's processors.

[lovromazgon]

Nice catch @hariso 👀 Sounds like a non-issue.

[simonl2002]

This is working as expected.