Capstone
Excalidraw AI: Text-to-Diagram Agent
Excalidraw is a free, open-source whiteboard for hand-drawn style diagrams. Because every Excalidraw drawing is plain JSON, a language model can write one: you type "draw the architecture of a RAG chatbot" and an agent returns an editable diagram. This page explains how text-to-diagram AI works with Excalidraw, shows the core of it in Python, and describes the capstone project AI Season students build in week 6.
By Abdul Rahman Azam, founder of AI Season · Updated
Key facts
| Excalidraw is | An open-source virtual whiteboard that saves drawings as JSON |
|---|---|
| Excalidraw AI means | Using a language model to generate or edit Excalidraw diagrams from a text prompt |
| Why it works | A diagram is a list of elements (rectangles, arrows, text) that a model can produce as structured output |
| Skills used | Structured output, tool calling, validation, layout logic, deployment |
| AI Season capstone | Session 11: an agent that turns a prompt into Excalidraw diagrams |
| Output | A .excalidraw file you can open and keep editing at excalidraw.com |
What is Excalidraw AI?
Excalidraw AI is the use of a language model to create diagrams in Excalidraw from a description in plain language, often called text to diagram. Excalidraw itself includes a text-to-diagram feature, and because the file format is open JSON, anyone can build their own generator with an LLM API.
The result is different from an AI-generated image. An image is pixels you cannot change. An Excalidraw diagram is a set of shapes, so you can drag a box, fix a label or add a step after the model has finished.
How text-to-diagram AI works
- The user describes the diagram: "a flowchart of how a tool-calling agent answers a question".
- The model plans the content as a simple graph: nodes with labels and edges between them.
- Code lays it out. A layout step assigns x and y positions so boxes do not overlap. Models are poor at arithmetic on coordinates, so this is better done in code.
- Code converts the graph to Excalidraw elements: one rectangle and one text element per node, one arrow per edge.
- Validate that the JSON matches the format and every arrow points to a node that exists.
- Return the file, or render it in an embedded Excalidraw canvas.
The key design decision: let the model decide what to draw, and let ordinary code decide where to draw it. Asking the model for raw pixel coordinates gives overlapping, crooked diagrams.
The Excalidraw file format in brief
An .excalidraw file is a JSON object with a type, a version and an elements array. Each element has a type (rectangle, ellipse, diamond, arrow, line, text), a position, a size and style fields.
{
"type": "excalidraw",
"version": 2,
"source": "my-diagram-agent",
"elements": [
{
"id": "node-user", "type": "rectangle",
"x": 100, "y": 100, "width": 200, "height": 70,
"strokeColor": "#1e1e1e", "backgroundColor": "#a5d8ff",
"fillStyle": "solid", "roughness": 1
},
{
"id": "label-user", "type": "text",
"x": 150, "y": 122, "text": "User question", "fontSize": 20
}
],
"appState": { "viewBackgroundColor": "#ffffff" },
"files": {}
}Real files carry more fields per element (seeds, versions, bindings). Excalidraw's open-source library includes a helper that expands a short element description into a full one, which is the easiest path in a web app.
A minimal diagram agent in Python
This script asks a model for a graph using structured output, lays the nodes out in a column, and writes a file you can open at excalidraw.com with File, Open.
import json
from openai import OpenAI
from pydantic import BaseModel
class Node(BaseModel):
id: str
label: str
class Edge(BaseModel):
source: str
target: str
class Diagram(BaseModel):
nodes: list[Node]
edges: list[Edge]
client = OpenAI()
def plan_diagram(prompt: str) -> Diagram:
completion = client.beta.chat.completions.parse(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Turn the request into a flowchart of at most 8 nodes. "
"Use short labels. Every edge must connect two existing node ids."},
{"role": "user", "content": prompt},
],
response_format=Diagram,
)
return completion.choices[0].message.parsed
def to_excalidraw(diagram: Diagram) -> dict:
width, height, gap = 240, 70, 60
position, elements = {}, []
for i, node in enumerate(diagram.nodes): # layout is done in code
x, y = 100, 100 + i * (height + gap)
position[node.id] = (x, y)
elements.append({"id": node.id, "type": "rectangle", "x": x, "y": y,
"width": width, "height": height, "strokeColor": "#1e1e1e",
"backgroundColor": "#a5d8ff", "fillStyle": "solid", "roughness": 1})
elements.append({"id": f"{node.id}-text", "type": "text", "x": x + 16, "y": y + 22,
"text": node.label, "fontSize": 20, "strokeColor": "#1e1e1e"})
for n, edge in enumerate(diagram.edges):
if edge.source not in position or edge.target not in position:
continue # validation: drop broken edges
sx, sy = position[edge.source]
tx, ty = position[edge.target]
start = (sx + width / 2, sy + height)
elements.append({"id": f"edge-{n}", "type": "arrow", "x": start[0], "y": start[1],
"points": [[0, 0], [tx - sx, ty - (sy + height)]],
"strokeColor": "#1e1e1e", "endArrowhead": "arrow"})
return {"type": "excalidraw", "version": 2, "source": "diagram-agent",
"elements": elements, "appState": {"viewBackgroundColor": "#ffffff"}, "files": {}}
diagram = plan_diagram("How a RAG chatbot answers a question from a PDF")
with open("diagram.excalidraw", "w") as f:
json.dump(to_excalidraw(diagram), f, indent=2)
print("Saved diagram.excalidraw")If the file does not open cleanly in your Excalidraw version, pass the elements through Excalidraw's own element-conversion helper in a small web page; it fills in the remaining fields. Field requirements have changed between releases, so check the current Excalidraw developer docs.
Turning the script into an agent
- Edit, not only create. Give the model tools such as
add_node,connect,renameanddelete_nodeso the user can say "add a cache between the API and the database". This is tool calling on a shared diagram state. - Better layout. Replace the single column with a layered layout so branches sit side by side.
- Self-check. Validate the output and send errors back to the model to fix, a small evaluator loop.
- Diagram types. Flowcharts, sequence diagrams, architecture diagrams and mind maps each need their own layout rule.
- A live canvas. Embed the open-source Excalidraw React component and update it as the agent works.
- Ship it. Put the agent behind an API and give it a public URL. See deploy an AI agent.
Excalidraw AI vs other text-to-diagram tools
| Approach | Output | Editable | Style |
|---|---|---|---|
| Excalidraw JSON | Shapes on a canvas | Yes, by drag and drop | Hand-drawn |
| Mermaid | Text syntax rendered as a diagram | Yes, by editing text | Clean, automatic layout |
| Graphviz DOT | Text syntax rendered as a graph | Yes, by editing text | Technical |
| Image generation model | A picture | No | Any, but text is often wrong |
A common trick is to have the model write Mermaid, which it does reliably, and then convert Mermaid to Excalidraw elements. Excalidraw supports that conversion for flowcharts.
The AI Season Excalidraw capstone
Session 11 of the AI Season curriculum is the Excalidraw Project: each student ships an agent that turns a prompt into structured Excalidraw diagrams, end to end. It was chosen as the capstone because it uses most of the course in one project:
| Course skill | Where it shows up in the capstone |
|---|---|
| Prompt engineering | A system prompt that produces a clean graph every time |
| Structured output and tool calling | The diagram schema and the edit tools |
| Design patterns | Plan, then draw, then validate and repair |
| LangGraph | State for the diagram across turns |
| Guardrails | Rejecting oversized or off-topic requests |
| Deployment | A public URL others can try |
The finished project is a visual portfolio piece: an interviewer or client can type a sentence and watch a diagram appear. See more AI agent project ideas. Cohort 02 starts 1st January 2027.
Frequently asked questions
Does Excalidraw have AI?
Yes. Excalidraw offers a text-to-diagram feature, and because its files are open JSON you can also generate diagrams with your own LLM agent.
How does text to diagram AI work?
A language model converts your description into a structured list of nodes and connections. Code then positions the shapes and writes them in a diagram format such as Excalidraw JSON or Mermaid.
Can ChatGPT or Claude create Excalidraw diagrams?
They can write Excalidraw JSON or Mermaid code that you import into Excalidraw. Results improve a lot when code handles the layout and the model only decides the content.
Is Excalidraw free?
The Excalidraw editor is free and open source. A paid plan adds collaboration and hosted features.
What do AI Season students build with Excalidraw?
The week 6 capstone is an agent that turns a text prompt into editable Excalidraw diagrams, combining structured output, tool calling, validation and deployment in one portfolio project.