[Idea] The Toolpad backend agent

[Janpot]

The Toolpad backend agent

Synopsis:

Our users want to interact with their own backend services. These services already have everything configured to interact with their downstream databases and APIs. Currently we rely on our users creating REST interfaces so that toolpad can interact with their backend. This requires a lot of ceremony. What if we allow our users to embed a small piece of toolpad directly in their service, where they can define queries usable in Toolpad.

Example:

Say the user has their existing node.js backend application

import express from 'express';

const app = express();

app.use('/foo', (req, res) => {
  // user code ...
});

app.listen(3001);

Note: express here is interchangeable with any framework that supports connect middleware. Or that have very straightforward adapters to such middleware. Which should cover most ground for Node.js users. For other languages we will have to come up with similar constructs.

They can embed the toolpad agent as follows:

import { ToolpadAgent } from 'toolpad';
import prisma from './db';

const agent = new ToolpadAgent({
  queries: {
    getEmployees: {
      params: {
        age: { type: number },
      },
      resolve(params) {
        // use ORMs or APIs already available in your backend
        return prisma.employees.findMany({ where: { age: paras.age } });
      },
    },
  },
  mutations: {
    // ...
  },
});

app.use('/toolpad', agent.handler);

This allows them to interface with any library they're already using in their backend, together with the secrets that are already available there.

After they deploy their backend service to production an endpoint will be available as https://user-domain/toolpad. They will need to expose this endpoint publicly so that Toolpad can connect to it.

The user will then go to the Toolpad application at e.g. app.toolpad.com and provide the url https://user-domain/toolpad to their project as a datasource. Now when creating queries based on this datasource, the user is presented with a dropdown containing 'getEmployees', ... When they select that item they get presented with bindable parameter age where they can connect page state.

During data fetching the query is proxied to the Toolpad agent in the user's backend and the result is made available on the page.

notes

• This is basically a trojan horse inside of the users backend. We'll need strong authentication mechanisms to protect this endpoint.
• We'll probably need to think about authentication and users/workspaces before this feature.
• We can create adapters for JS frameworks with minimal code, e.g. Next.js accepts standard connect handlers for API routes
• We can create similar libraries for different programming languages
• This is a bit similar to openAPI, maybe we should just start with an openAPI datasource first?

[apedroferreira]

I like this feature, it would have many advantages compared to function datasources such as being more customizable, not having to store your private keys in Toolpad and being able to run complex queries without adding too much load to our servers.
But I wonder how much easier it would be to use the Toolpad agent than to just create REST API endpoints that did the same thing?

[Janpot]

But I wonder how much easier it would be to use the Toolpad agent than to just create REST API endpoints that did the same thing?

• REST API endpoints don't allow for introspection. With the Toolpad agent we control the full endpoint. We can add a way for Toolpad to introspect all the methods available, and parameters. That way Toolpad can show richer UI to the user when creating queries.
• Error handling will be more straightforward
• The user shouldn't need to think about the transport method, it's an implementation detail.
• We can build authentication in the agent and tie it to toolpad authentication. NO separate mechanisms required.
• ...

[apedroferreira]

Those are good points, and we could extend as much as we wanted to add even more functionality.
So it sounds really interesting!

[oliviertassinari]

Interesting idea, it sounds like a way to solve this problem: generating backend APIs in a format that Toolpad can directly interface with. I would imagine that on this trend, we could look into a Swagger integration.

On a different note, I wonder if we couldn't go one step lower in abstraction by making it possible to provide functions like Next.js's API https://beta.nextjs.org/docs/data-fetching/api-routes, not even an express server, Toolpad host it. Or like https://docs.airplane.dev/tasks/js. The value is that it comes with built-in permissions and execution logs. This would make it possible to host this script: https://github.com/mui/mui-store/blob/master/accounting/generatePayout.mjs.

[Janpot]

not even an express server, Toolpad host it

The idea here is that you already have an express-like server, you're just integrating a toolpad component into it. The advantage is that you already have everything at the tips of your fingers there, your ORM, db connection, stripe API, secrets,... Having Toolpad host the backend will require you to set all that stuff up again, but on Toolpad. And to keep the config synced as your backend changes. Having Toolpad explicitly not host it would be a feature IMO.

I believe that if we want to make it easier to integrate your backend with Toolpad, we should aim for a solution that

• minimizes the amount of secrets that have to be duplicated from your backend to your Toolpad server
• requires minimal migration efforts of the Toolpad application as your backend evolves. avoids having to sync up deploying a breaking change on your backend with deploying a change on the Toolpad application.
• minimizes the amount of code/configuration that deals with transport logic. I should just write a function and its output should become available on Toolpad.

So far, direction 12 is the one that tries to get closest to a workflow like that

[oliviertassinari]

I imagine these are not mutually exclusive? We could do both

• For the generatePayout.mjs use that I have, I would rather not create a backend, it's a pain.
• For somebody that already has a server, I would imagine that the current pains are: 1. How do I create an API route that has the right permissions, I will already configure Toolpad with the ACL logic? I would imagine that today's solution with Retool is to have something based on authenticated Retool and give a wildcard. 2. How do minimize the level of duplication of work, e.g. why should I recreate a query with parameters in Toolpad, when I already have to create it server-side?

[Janpot]

The way I'd imagine this is we could say from Toolpad side, "here's a secret key", configure it in your backend, we'll handle the authentication.