An SDK on top of our existing workflow and orchestration tooling, simplifying and improving the developer experience of managing workflows.
In the example below, we define a simple workflow with two steps.
import { createWorkflow, createWorkflowHandler, createTransformer } from "@medusajs/workflows"
const myHandler1 = createWorkflowHandler("first", myHandler1Invoke, myHandler1Compensate)
const myHandler2 = createWorkflowHandler("second", myHandler2Invoke, { saveResponse: false })
const myWorkflow = createWorkflow("myWorkflow", async ({ data, container, context }) => {
const first = await myHandler1(data)
await myHandler2(first)
})
const input = { data: "someData" }
const container = {}
const context = { manager: "manager" }
myWorkflow.run(input, container, context)
This example illustrates a few key points:
- Little to no boilerplate code is needed
- Workflows are created using simple JavaScript
- Data can be passed from one function to another as you would normally do
Obviously, this is a very simple example. However, it should be clear that the SDK will eliminate a significant amount of boilerplate code and provide a more intuitive API to manage workflows.
New APIs in the proposal
createWorkflow
: Create a workflow and build its step definitioncreateWorkflowStep
: Create a workflow stepreplaceAction
: Replace a workflow step
This proposal comes with a new createWorkflow
util:
type MyWorkflowInput = {}
type MyWorkflowOutput = {}
const myWorkflow = createWorkflow<MyWorkflowInput, MyWorkflowOutput>(
"myWorkflow",
async ({ data, container, context }) => {
const first = await myHandler1(data)
await myHandler2(first)
}
)
The createWorkflow
util is responsible for creating the workflow and building its step definition.
In the function body, we scan for handlers created using the createWorkflowStep
util and register them as steps in the definition.
Additionally in this "registration step", data can be passed between handlers. The functionality is similar to that of the pipe
function in existing workflows.
In the first iteration, the registration step is limited to the extend that it only scans for handlers, so all other logic in the function is ignored and without effect. However, one could imagine we could eventually support any type of JavaScript code, for example if-then statements. See Discussion at the bottom of the proposal.
Under the hood, the createWorkflow
util creates all necessary boilerplate code:
export const myWorkflowName = "myWorkflow"
export const steps: TransactionStepsDefinition = {}
export const handlers = new Map([])
WorkflowManager.register(myWorkflowName, steps, handlers)
export const myWorkflow = exportWorkflow(myWorkflowName)
After having created the workflow, you add workflow steps using a new util createWorkflowStep
:
import { createWorkflowStep, createWorkflow } from "@medusajs/workflows"
type Handler1Input = {}
type Handler1Output = {}
const myHandler1 = createWorkflowStep<Handler1Input, Handler1Output>(
"first",
myHandler1Invoke,
myHandler1Compensate
)
const myWorkflow = createWorkflow("myWorkflow", async ({ data, context }) => {
const first = await myHandler1(data)
})
As seen in the snippet above, the util accepts the following:
first
-> workflow step namemyHandler1Invoke
-> invoke function of the stepmyHandler1Compensate
-> compensate function of the step
An alternative method signature of the util could look like:
const myHandler1 = createWorkflowStep<Handler1Input, Handler1Output>("first", {
invoke: myHandler1Invoke,
compensating: myHandler1Compensate,
onComplete: () => console.log("hello world")
})
The alternative would allow for more options without having to juggle correctly positioned method arguments.
Under the hood, the createWorkflowStep
will extend the workflow definition and handlers as follows:
export const steps: TransactionStepsDefinition = {
next: {
action: "first"
}
}
export const handlers = new Map(
[
"first",
{
invoke: myHandler1Invoke,
compensate: myHandler2Compensate
}
]
)
In the example above, the pipe
function is left out for the sake of simplicity.
Adding another workflow step
type Handler2Input = {}
type Handler2Output = {}
const myHandler2 = createWorkflowStep<Handler2Input, Handler2Output>(
"second",
myHandler2Invoke,
{ maxRetries: 3, retryInterval: 1000 }
)
const myWorkflow = createWorkflow("myWorkflow", async ({ data, container, context }) => {
const first = await myHandler1(data)
await myHandler2(first)
})
Under the hood
export const steps: TransactionStepsDefinition = {
next: {
action: "first",
next: {
action: "second",
noCompensation: true, // handler was created without a compensating action
maxRetries: 3,
retryInterval: 1000
}
}
}
export const handlers = new Map([
[
"first",
{
invoke: myHandler1Invoke,
compensate: myHandler2Compensate
}
],
[
"second",
{
invoke: myHandler2Invoke,
}
],
])
To transform data between workflow steps, we will use the same createWorkflowHandler
utility to create a new step in the workflow definition. The goal is to make the SDK so intuitive and simple so that you only ever need to understand the concept of a step.
Until now, using steps for transformation, validation, and core handler logic would result in an unmaintainably long definition. To solve this, we chose to allow for multiple handlers in a single step composed using the pipe
function. This is all good and fine.
However, abstracting the handlers definitions and providing a simple API allows us to use steps for all business logic without having to worry about maintainability, readability, etc.
Overall, this will make workflows more composable.
const myTransformer = createWorkflowStep("myTransformer", myTransformer)
const myWorkflow = createWorkflow("myWorkflow", async ({ data, container, context }) => {
const first = await myHandler1(data)
const transformedData = await myTransformer(first)
await myHandler2(first)
})
Under the hood
export const steps: TransactionStepsDefinition = {
next: {
action: "first",
next: {
action: "myTransformer",
next: {
action: "second",
noCompensation: true, // handler was created without a compensating action
maxRetries: 3,
}
},
}
}
export const handlers = new Map([
[
"first",
{
invoke: myHandler1Invoke,
compensate: myHandler2Compensate
}
],
[
"myTransformer",
{
invoke: myTransformer,
}
],
[
"second",
{
invoke: myHandler2Invoke,
}
],
])
To extend a workflow, you will use a combination of the already presented createWorkflowStep
util and the existing appendAction
method of the WorkflowManager
.
const myHandler3 = createWorkflowStep("third", myHandler3Invoke, { saveResponse: false })
myWorkflow.appendAction("third", "second", myHandler3)
Under the hood
export const steps: TransactionStepsDefinition = {
next: {
action: "first",
next: {
action: "myTransformer",
next: {
action: "second",
noCompensation: true, // handler was created without a compensating action
maxRetries: 3,
next: {
action: "third",
noCompensation: true, // handler was created without a compensating action
saveResponse: false,
}
}
},
}
}
export const handlers = new Map([
[
"first",
{
invoke: myHandler1Invoke,
compensate: myHandler2Compensate
}
],
[
"myTransformer",
{
invoke: myTransformer,
}
],
[
"second",
{
invoke: myHandler2Invoke,
}
],
[
"third",
{
invoke: myHandler3Invoke,
}
],
])
To replace a step in the workflow, you'll again use the util createWorkflowStep
in combination with the replaceAction
of the WorkflowManager
.
You cannot use appendAction
to replace a step.
Incorrect
const myNewHandler2 = createWorkflowStep("second", myHandler2Invoke)
myWorkflow.appendAction("first", "second", myNewHandler2)
Correct
const myNewHandler2 = createWorkflowHandler("second", myHandler2Invoke)
myWorkflow.replaceAction("second", myNewHandler2)
Can we support all JavaScript, e.g. if-else, in the workflow build step?
const myWorkflow = createWorkflow("myWorkflow", async ({ data, container, context }) => {
const first = await myHandler1(data)
if (!first.someArray.length) {
someHandler()
} else {
await myHandler2(first)
}
})
Unrelated to the Workflow SDK
- Where do we place custom workflows in a Medusa project?
Following our established patterns, one solution would be to use the file system with a
/workflows
directory and add a related loader. - Where do we place workflow overrides in a Medusa project?
Following our established patterns, one solution would be to use the file system with a
/workflows
directory and add a related loader. - Where do we place workflow modifications in a Medusa project? This is different from a loading mechanism, as suggested above, so we might need to come up with a new pattern.
Overhaul It looks very nice and simple, of course, as mention in the gist, some configurations etc are left aside for simplicity.
I mentionned to seb that it would be nice to have a skip option which would be used t skip a step based on more global rules, in opposition to the if-then, both are valid and I think both could be available. My thinking is more about the finale workflow representation, let say in the UI. having a skip based on global things such as feature flag would allow us to display to the user what steps will run and which one will be skipped. When it comes to if-then it really depends on the data and we can't give the information the user before we get to the data and in that case it is managed by the step handler. Though, if the if-then was extracted somewhere else we could display to the user what are the condition for a step to run one handler or another. I would say that in that case both handlers in the step example should be part of two different step that run under certain condition. It would allow us to display the execution constraint to the user instead of having them hidden in the step hanndler upon creation. For example, if a user want to extend a workflow in a visual fashion we would be able to show all the necessary information.
About workflow extension, one thing that we will have to tackle is that the user has no idea if the workflow has already been extended or not when adding a new step. Therefor the complexity could be that the user add a new step but the step end up not in the right place if it is extended elsewhere during development, have you thought about something to resolve that? or maybe this is not something to resolve and the user just need to read the documentation of what he is installing.