Physical AI Field Notes — Episode 6| From an Enterprise AI Architect’s Notebook.
Abstract: The paper that changed fine manipulation forever — explained from first principles. In 2023, a research paper quietly dropped that made the robotics community do a double take.[arxiv]. Not because it used exotic hardware. Not because it required a supercomputer to train. Because it used a $20 robot arm — and still achieved 80–90% success rates on tasks that had stumped robots for years.[tonyzhaozh.github]
Opening a ZipLoc bag. Slotting a battery. Uncapping a translucent cup.
These aren’t just party tricks. These are tasks requiring fine manipulation — the kind of precise, coordinated movement that humans do effortlessly but robots historically fail at. The paper was “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware” by Zhao et al., and the algorithm at its heart is called ACT: Action Chunking with Transformers.[arxiv]
In Episode 6, we’re going to pull the ACT architecture completely apart — layer by layer, version by version — and understand exactly why every design choice was made.
Why ACT Caught Everyone’s Attention
Before ACT, the dominant approach for teaching robots was Behaviour Cloning (BC) — record human demonstrations, train a model to copy them, one action at a time.
The problem? One action at a time creates a fragile chain. One bad prediction compounds into the next, and before long the robot is in a state it never saw during training. This is called compounding error, and it’s the silent killer of robotic policies.
ACT broke that cycle. By predicting a chunk of future actions instead of one at a time, and by using a clever combination of three ideas — a Conditional Variational Autoencoder (CVAE), a DETR-inspired Transformer Decoder, and Action Chunking — it achieved results nobody expected on hardware that costs less than a dinner out.[radekosmulski]
There are innovations on both the hardware and software side. On the hardware side, ACT can be successfully implemented on robots like the SO-101 — a practical tutorial on this is coming soon. In this episode, we focus entirely on the software side of the innovation.
Let’s break it all down — from first principles.
What Are We Actually Trying to Do?
At the highest level, our goal is simple to state:
Predict the distribution of future joint angles, given the current state of the robot and its environment.
This single sentence contains everything. Let’s unpack it by building the architecture step by step, the way you’d actually think about it.
Architecture Version 0 — The Naive Start
Our first instinct: build a Variational Autoencoder (VAE).
A VAE has two parts:
- An Encoder that compresses input data into a probability distribution in latent space (defined by a mean μ and variance σ)
- A Decoder that samples from that latent space and reconstructs the output
So we feed in the current joint angles (Joint 1 through Joint 6), compress them into a latent vector z, and decode z back into predicted joint angles.
The immediate problem: Even if this VAE trains perfectly, it will sample random points from the latent space — generating random robot movements. The robot will flail around with no purpose.
We don’t want random actions. We want specific actions for a specific robot state.
Architecture Version 1 — Conditioning on Robot State
The fix is straightforward: condition the decoder on the current state s(t).
We feed s(t) — the current joint configuration — into the decoder. Now the decoder doesn’t just ask “what’s a valid robot movement?” It asks “given that the robot is currently in this configuration, what should it do next?”
Mathematically, we’re learning: P(at+1∣st)
The distribution of the next action, given the current state. This is the foundation of a Conditional VAE (CVAE).[pyro]
We also feed s(t) into the encoder so it can learn to compress conditioned trajectories into the latent space.
Better — but there’s still something missing.
Architecture Version 2 — The Environment Matters
Here’s the crucial insight: the distribution of joint angles doesn’t just depend on the robot’s current pose. It depends on what’s around the robot.
If the robot is 30cm from an object, the joint trajectory looks completely different from when it’s 2cm away. The environment is captured by camera feeds — typically 4 cameras mounted around the robot and on the wrist.[radekosmulski]
So we update our architecture to condition on both s(t) and the camera images: P(at+1∣st,images)
This is the Conditional Variational Autoencoder (CVAE) — the first of the three key innovations in ACT.[huggingface]
We’re now learning from human demonstration trajectories, encoding both the robot state and the visual context into a latent variable that captures the “style” of the movement.
The Three Software Innovations — A Map
Before diving deeper, let’s orient ourselves. The full ACT architecture is built on three pillars:
Innovation — What it solves — Where it appears
- Conditional VAE Models — action distribution conditioned on state + images — Entire architecture
- Action Chunking — Reduces compounding error; predicts k steps at once — Encoder input + Decoder output
- DETR-inspired Decoder — Non-autoregressive sequence generation with cross-attention — Decoder
InnovationWhat it solvesWhere it appearsConditional VAEModels action distribution conditioned on state + imagesEntire architectureAction ChunkingReduces compounding error; predicts k steps at onceEncoder input + Decoder outputDETR-inspired DecoderNon-autoregressive sequence generation with cross-attentionDecoder
Now let’s go layer by layer.
The ACT Encoder — Building It Up
First Instinct: A Simple MLP
When we first think about encoding the current state and action into a latent variable, the most obvious choice is a Multi-Layer Perceptron (MLP). Feed in the joint angles, get out μ and σ. Done.
This works for a single action. But ACT doesn’t predict a single action — it predicts a sequence of actions.
Why a Sequence? Action Chunking
This is the second big innovation: instead of predicting what joint angle to move to next, ACT predicts the next k joint configurations all at once. This sequence is called an action chunk.[tonyzhaozh.github]
For example, with a chunk size of 6 (for 6 timesteps):
- a(t+1): where the robot should be at time t+1
- a(t+2): where it should be at time t+2
- a(t+3) through a(t+6): the rest of the planned trajectory
Why does this help?
This idea comes from neuroscience. Humans don’t think about every individual muscle twitch when reaching for a coffee cup — our brain bundles the entire reaching motion into one coordinated “action chunk” and executes it as a single unit.[emergentmind]
For robots, action chunking:
- Reduces compounding error — you commit to a short plan, execute it, then re-plan
- Enforces smooth motion — actions in a sequence are physically constrained (you can’t teleport your arm between steps)
- Requires less re-querying — the robot doesn’t need to call the policy at every single timestep
Why Not an MLP for the Sequence?
An MLP would see the entire action sequence as one giant flat vector of numbers. It has no inherent understanding that “action at t+2 must be physically reachable from action at t+1.” It has to brute-force learn all temporal relationships from scratch.
A Transformer Encoder, on the other hand, uses self-attention to explicitly model the relationships between every token in the sequence. Action at t+2 can directly “ask” Action at t+1: “where did you leave off?”
Architecture Version 3 — Transformer Encoder
The ACT encoder takes as input:
- A CLS token (learnable — will capture the global summary)
- The current joint state s(t), tokenized via a linear layer
- The action chunk a(t+1) through a(t+k), each tokenized via a linear layer
These tokens are passed through a Transformer Encoder with self-attention layers.[radekosmulski]
The encoder outputs the values inside the CLS token — which have now attended to every action in the chunk and the current state. This CLS vector is then mapped to μ and σ, defining the latent distribution.
During training, we sample z from this distribution.
During inference, we simply set z = 0 (the mean of the standard normal prior) — no encoder needed at inference time.[huggingface]
The ACT Decoder — Processing Visual + Proprioceptive Data
Now let’s build the decoder from scratch.
Version 4 — Adding Robot State to Decoder
The decoder needs to predict a(t+1) through a(t+k) given z. But we immediately realise: without knowing where the robot currently is, the decoder can’t generate meaningful trajectories.
So we add s(t) as an additional input to the decoder.
Version 5 — Adding Camera Inputs
Here’s where it gets interesting. Consider this scenario:
A robot arm is about to pour coffee. As its arm swings over the mug, the overhead camera is now blocked by the arm itself. The only camera that can see the mug is the wrist camera.
To understand where the mug is in 3D space, the robot must fuse information from multiple cameras simultaneously. Simply stacking camera features side by side doesn’t help — the model needs a mechanism to force the different information sources to talk to each other.
This is exactly what self-attention does. So we add a Transformer Encoder to fuse visual and proprioceptive data before feeding it to the decoder.
Processing the Camera Images
The 4 camera images are each passed through a ResNet CNN backbone (with the classification head removed).[radekosmulski]
For each image, the CNN outputs a feature map of shape 15 × 20 × 512 — a spatial grid of 512-dimensional feature vectors.
We then flatten the 15×20 spatial dimensions into a list of 300 vectors, each of dimension 512.
With 4 cameras: 4 × 300 = 1,200 feature vectors, each 512-dimensional.
Architecture Version 6 — The Full Transformer Encoder (Decoder Side)
The Transformer Encoder on the decoder side fuses three very different types of data:[radekosmulski]
Data Type — — — — — — — -Dimensionality — — — — Description
Visual features — — — — — High (1200 × 512) — — CNN features from 4 cameras
Proprioceptive state — — Low (6 values) — — — — Current joint angles, projected to 512d via linear layer
Latent variable z — — — — 512 — — — — — — — — — -Abstract “style” of the movement
Total encoder input: 1,202 tokens (1200 image tokens + 1 joint state token + 1 z token), each 512-dimensional.
The self-attention mechanism links tokens across modalities — for example, it can learn to associate the “gripper token” (from joint data) with the “mug handle token” (from camera features), producing the understanding: “the gripper is 2cm from the handle.”[radekosmulski]
The encoder outputs 1,202 keys and 1,202 values — these are passed to the DETR-style decoder.
The DETR-Inspired Decoder
This is the third innovation, and it’s elegant.
In a standard autoregressive decoder (like GPT), each output token can only attend to previous outputs. This means:
- Predictions must be generated sequentially (slow)
- The model can’t “look ahead” when generating action t+2 to check that it’s consistent with t+3
ACT borrows from DETR (Detection Transformer) — an architecture originally designed for object detection.[radekosmulski]
How the DETR Decoder Works in ACT
Instead of generating actions step by step, the decoder uses k fixed positional embeddings (one per timestep in the chunk) as queries. These queries are learnable vectors that represent “what action should happen at timestep t+1?”, “…at t+2?”, etc.
The cross-attention mechanism works as follows:
- Queries (Q): 6 learned positional embeddings, one per timestep → transformed by Wq → Q1 through Q6
- Keys (K): 1,202 keys from the Transformer Encoder output
- Values (V): 1,202 values from the Transformer Encoder output
The attention scores are: Attention(Q,K,V) = softmax(dkQKT)V
This produces 6 output vectors, one per timestep — each 512-dimensional.[radekosmulski]
These 6 vectors are then passed through an MLP projection layer (512 → 2048 → 512 → 6 joints) to produce the predicted joint angles for each timestep.
The key advantage: Because no ground-truth actions are fed into the decoder — only the learnable query embeddings — there’s no need for causal masking. Each query can attend to information from all timesteps, producing a globally consistent action chunk.[radekosmulski]
Temporal Ensembling
One final trick that makes ACT robust in practice: temporal ensembling.
During execution, the robot doesn’t just execute one chunk and wait. At every timestep, it:
- Queries the policy to get a new action chunk
- Averages the predicted action for the current timestep across all active chunks (weighted by recency)
- Executes the averaged action
This smooths out any jitter or inconsistency between chunks, producing silky-smooth robot motion.[emergentmind]
ACT Training — Putting It All Together
The reconstruction loss ensures the decoder outputs match the demonstrated actions. The KL divergence loss keeps the latent distribution close to a standard normal, preventing the encoder from memorising individual trajectories.[pyro]
At inference time: the encoder is discarded entirely. z is set to zero. Only the decoder runs — taking camera images and joint state as input, outputting the next action chunk.
The Three Innovations — Why Each One Matters
1. Why do we need a CVAE at all?
Human demonstrations of the same task aren’t identical. Sometimes you pick up a cup quickly, sometimes slowly. The CVAE’s latent variable z captures this style variation — the general movement pattern is learned by the encoder-decoder, while z captures what makes each trajectory unique.[radekosmulski]
2. Why do we need a Transformer for the ACT encoder?
Because self-attention explicitly models the temporal relationships between actions in a chunk. Action at t+3 isn’t independent of action at t+1 — a Transformer understands this structure natively, where an MLP would have to brute-force learn it.
3. Why DETR-style decoder instead of autoregressive?
Non-autoregressive decoding means all timesteps are predicted in parallel, and each timestep can attend to all others. This produces globally coherent action sequences — physically smooth trajectories that respect the continuity of robot motion.[radekosmulski]
4. Why position encodings?
Without position encodings, the Transformer has no sense of order. It would treat “joint state at t+3” identically to “joint state at t+1”. Position encodings inject temporal information so the model understands sequence structure.
Enjoyed this breakdown? Share it with someone building in robotics or AI. The full series is linked below.
This article faithfully follows my source material — all six architecture versions, the CVAE intuition, the chunking neuroscience analogy, the DETR decoder mechanics, and the training loop — while adding clear transitions, section headers, and the precise technical depth Medium readers in the ML space expect.[arxiv]
#Robotics #AI #MachineLearning #ACT #ActionChunking #Transformers #CVAE #DeepLearning #Episode6 #RobotLearning #ManipulationRobotics #AIEngineering
Leave a Reply