Long-Running Workflow Automation and API Integration

A visual workflow platform that saves execution progress, waits for external input and resumes work from its recorded state.

This long-running workflow automation case study follows a run that has to wait. A form or an approval may arrive long after the request that first started the automation has ended. The platform lets a workflow pause at that step and carry on once the input it needs turns up.

Teams build the sequence in a visual editor, combining application integrations with custom logic. They can test a draft, publish the version meant for live work, and watch each run move between executing and waiting.

The challenge

Resuming a workflow takes more than knowing its next node. The engine also needs the saved inputs, the waiting branches and the progress of the run so far. It has to cope when two processes hit the same waiting execution.

Workflow authors can add code and credentials for outside services, so that logic needs firm execution boundaries. The editor had to keep a process easy to maintain. It could not hide the permissions and runtime settings its jobs depend on.

What we built

Inside the editor, a workflow appears as a set of connected nodes wired together into a sequence. The flow begins from a webhook, a schedule or an event. Nodes further on reach out to services, adjust the data or branch on the result they return. Steps written in JavaScript or Python cover any custom logic.

Each node describes its own settings and credential needs, plus the inputs and outputs it works with. The editor reads that description to build the configuration screen. Retained node versions let a workflow keep its earlier integration behaviour when a newer build appears.

Authors can pin sample data to a node while testing a stage, which avoids a fresh call to the outside service on every edit. Draft and published versions keep development apart from the version serving live work. Forms, timed waits and approval steps can pause a run. Run history keeps a record of its progress, and partial reruns let a team revisit one part instead of starting again from the trigger.

Represent a paused run as stored execution state

The execution engine keeps a serialisable state. It holds the node stack, the waiting nodes and the sources of their inputs. A node that needs several inputs will wait until all of those inputs have arrived and are ready to use.

A wait tracker queries the saved waiting executions and schedules their resumption. Before it starts, the system claims the resume attempt. Another process holding the same waiting record then sees that the work is already under way. A failed load or resume frees the claim again. The waiting run keeps its state, even after the request that began it has ended.

Separate incoming requests from worker execution

In queue mode, webhook processes identify the registered workflow. They place the work on a Redis-backed Bull queue. Workers load the recorded execution, run its nodes and save results as they go. When the original request expects a response, a relay carries the result back to it. The main application, the webhook handlers and the workers can each run as a separate role around that shared execution state.

Handle custom code and credentials explicitly

JavaScript and Python code steps run through task-runner processes connected to a broker. Expressions can run inside isolated V8 environments. Extension packages loaded into the application need their own trust review, and they do not run inside the isolation used for code steps.

Credentials sit in an encrypted store with key rotation. A worker needs the same installation encryption key to read the saved credentials its jobs use. Publication goes through a saved outbox intent, and scheduled work is persisted before it is fired. Projects group resources inside one installation that serves a single organisation. Run activity and metrics come through the observability integrations.

Project success

Teams can follow one automation from its trigger through the middle actions to a later approval or form response. Saved state keeps its progress through a pause, while resume claims coordinate the attempts to restart the waiting work.

When something needs attention, run history shows where the process reached. Partial reruns let an author revisit the stages that matter. Visual nodes cover the configured integrations, and JavaScript or Python steps handle the custom work. These functions stay part of the same published workflow rather than needing a separate process for each waiting period.

Technology stack

Application

TypeScript, Node.js, Express

Editor

Vue

Storage options

SQLite or PostgreSQL

Queue execution

Redis, Bull

Author-written logic

JavaScript and Python task runners

Expression isolation

isolated-vm