For years, much of artificial intelligence was built around a relatively comfortable architecture:

data
  ↓
cloud
  ↓
model
  ↓
response

That pattern works fairly well when a response can take a few hundred milliseconds, when the Internet is available, and when an error ends on a screen.

The physical world changes the rules.

A robotic arm, autonomous vehicle, industrial machine, or inspection system cannot assume there will always be a perfect connection to a data center. Nor can it wait indefinitely for a remote model to decide what to do while the environment keeps moving.

That is where Physical AI appears: artificial-intelligence systems capable of perceiving, deciding, and acting in the real world.

A recent TechRadar Pro article, “How to deploy physical AI effectively”, summarizes three requirements that sound simple but completely change the architecture:

  • very low latency;
  • offline-first operation;
  • efficient compute and storage near the point of action.

The deeper conclusion matters:

Physical AI is not cloud AI stuffed inside a robot.

It needs a different distribution of responsibilities across edge, cloud, simulation, training, data, evaluation, and deployment.

And one of the most interesting repositories for studying how that stack is beginning to materialize is microsoft/physical-ai-toolchain, an open-source project connecting Azure with several parts of NVIDIA’s Physical AI ecosystem and robotics tools such as ROS 2 and LeRobot.

It is not an SDK for moving a servo.

It is something more ambitious: an attempt to turn the complete learning lifecycle of a robot into a reproducible platform.

Physical AI has two loops, not one

A useful way to understand the problem is to separate two cycles that operate at completely different speeds.

The first is the physical control loop.

sensors
   ↓
perception / state
   ↓
policy / inference
   ↓
control
   ↓
actuators
   ↓
physical world
   └──────────────→ new sensor data

That loop has hard constraints.

It may execute tens or hundreds of times per second. If the network disappears, the robot cannot freeze while waiting for an API. If a camera produces a new frame, the system needs to process it within a known timing budget.

That is why inference that affects motion usually lives at the edge, close to sensors and actuators.

But there is a second, much slower cycle: the learning loop.

robot
  ↓ captures experience
ROS 2 bags / datasets
  ↓
curation and validation
  ↓
training
  ↓
evaluation
  ↓
model registry
  ↓
packaging
  ↓
deployment
  ↓
robot

That second loop can take advantage of remote GPUs, cloud storage, simulators, experiment tracking, and orchestration systems.

This separation explains why a serious Physical AI architecture cannot be summarized as “put an LLM on a Jetson.”

The real problem is building the factory that produces, validates, and distributes the policies that later execute locally.

That is precisely what Microsoft Physical AI Toolchain tries to structure.

What microsoft/physical-ai-toolchain actually is

The README describes it as an open, production-oriented framework integrating Microsoft Azure services with NVIDIA’s Physical AI stack.

The architecture covers eight domains:

Infrastructure
Data Pipeline
Data Management
Synthetic Data
Training
Evaluation
Fleet Delivery
Fleet Intelligence

Importantly, the repository no longer presents all of those components as one monolithic requirement.

Microsoft reorganized the project around a T0 → T5 adoption ladder.

T0 Dev
│  1 robot + laptop
│  zero cloud / zero Kubernetes
│
T1 Lab
│  few robots + shared storage
│
T2 Pilot
│  Azure ML / OSMO / registry / MLflow
│  recommended for production
│
T3 Production
│  k3s + FluxCD at one site
│
T4 Scale
│  multiple sites + Azure Arc + GitOps + gating
│
T5 Operate
   fleet intelligence / drift / retraining
   roadmap

This architectural decision is probably one of the best ideas in the repository.

The project explicitly recognizes that forcing a team to deploy Azure Arc, AKS, Flux, IoT Operations, and the entire cloud stack before getting one robot to learn one task would be a poor experience.

At T0, the whole loop can be closed with one robot and one laptop.

No Kubernetes.

No Azure.

No Arc.

That lets teams start with the robotics problem and add infrastructure only when scale creates a real need for it.

T0 shows which pieces are truly essential

Tier 0 is especially educational because it strips the system down to its minimum components.

The recommended flow is approximately:

1. capture demonstrations with ROS 2
2. copy the data to the laptop
3. inspect / curate the dataset
4. train a LeRobot policy
5. evaluate locally
6. run the policy on the robot again

For data capture, the repository uses ROS 2 bags.

A minimal example from the documentation is:

ros2 bag record -o demos/insertion-task /observations /actions

In more realistic configurations, the system can record topics such as:

/joint_states
/camera/color/image_raw
/imu/data

Capture configuration can set frequency and compression per topic. The documentation uses, for example, lz4 for high-frequency numeric telemetry and zstd for images when stronger compression matters.

That detail matters much more than it seems.

A robot can produce enormous amounts of information. Cameras at 30 FPS, joints at 100 Hz, IMUs at 200 Hz, and other sensors begin competing for I/O, storage, and bandwidth long before training even enters the conversation.

Physical AI starts as real-time data engineering too.

From ROS bag to a dataset the model understands

Capturing data does not mean we are ready to train.

The repository uses LeRobot, Hugging Face’s robotics-learning project, as one of its primary representations for imitation learning.

The storage architecture distinguishes raw data from converted datasets.

Conceptually:

ROS 2 / MCAP
   ↓
conversion
   ↓
LeRobot dataset
   ├── meta/info.json
   ├── meta/stats.json
   ├── data/*.parquet
   └── videos/*.mp4

In the Azure Storage design, original bags live under paths such as:

datasets/raw/{device-id}/{date}/episode.mcap

while processed versions move into something like:

datasets/converted/{dataset-id}/

This introduces a foundational property for physical systems: the model must be traceable back to the data that produced it.

If a new policy starts behaving badly, we need to know:

  • which episodes entered the dataset;
  • which converter version was used;
  • which normalization statistics were applied;
  • which code trained the model;
  • which checkpoint ended up deployed.

In a chatbot, a regression can produce a worse answer.

In a robot, a regression can produce physical movement.

Traceability stops being a convenience and becomes part of the safety story.

Three training families: RL, IL, and VLA

The repository’s training/ domain separates three approaches:

training/
├── rl/     reinforcement learning
├── il/     imitation learning
└── vla/    vision-language-action

Reinforcement Learning

The RL path relies mainly on NVIDIA Isaac Lab.

Here the robot learns policies through interaction with simulated environments and reward functions. Instead of relying only on human demonstrations, we can run thousands or millions of episodes in simulation.

Imitation Learning

The IL path uses LeRobot and demonstration datasets.

The idea is straightforward:

human performs task
        ↓
robot observes state + action
        ↓
dataset
        ↓
policy learns to imitate

The repository includes pipelines for training policies such as ACT — Action Chunking with Transformers.

Vision-Language-Action

The third path is VLA: models connecting visual perception, language instructions, and robotic actions.

Project workflows include support for fine-tuning NVIDIA GR00T models in addition to traditional Isaac Lab and LeRobot pipelines.

This shows where the industry is heading.

We do not only want a model that learns one specific trajectory.

We want robots capable of receiving a more general intention and turning it into action sequences.

OSMO turns training into a distributed workload

When one laptop is no longer enough, NVIDIA OSMO appears.

OSMO acts as an orchestration layer for simulation and training workloads. The repository includes templates for jobs such as:

train.yaml             → Isaac Lab RL
lerobot-train.yaml     → LeRobot imitation learning
groot-train.yaml       → GR00T VLA fine-tuning

At T2, Azure ML and OSMO move work onto GPU clusters, run distributed training, store checkpoints, and connect runs to MLflow.

Here another difference from conventional AI applications appears.

A robotics training job does not simply produce one final file.

It produces a chain of artifacts:

versioned dataset
   ↓
configuration
   ↓
run / experiment
   ↓
checkpoints
   ↓
metrics
   ↓
registered model

The repository even requires Azure ML data assets to be referenced with explicit versions —for example azureml:name:3— and rejects shortcuts such as @latest in order to preserve reproducibility.

That is a small decision with the right philosophy:

a robot should not change behavior because “latest” meant something different this morning.

Simulation: manufacture experiences before risking hardware

One of Physical AI’s most powerful properties is that much learning can happen before touching the real world.

The toolchain integrates Isaac Sim / Isaac Lab and also has a dedicated synthetic data domain based on NVIDIA Cosmos.

The pipeline described by Microsoft chains three capabilities:

simulation
   ↓
Cosmos Transfer 2.5
   ↓ photorealism
Cosmos Predict 2.5
   ↓ plausible futures
Cosmos Reason 2
   ↓ evaluation / curation
training data

The underlying idea is to address one of robotics’ largest problems: obtaining enough representative experiences.

In the real world, collecting a new situation can require hours of operation, staff, hardware, and physical risk.

In simulation we can vary:

  • lighting;
  • object positions;
  • textures;
  • cameras;
  • friction;
  • geometry;
  • failures;
  • rare scenarios.

The goal is not to make simulation perfectly identical to reality.

It is to generate enough diversity for the policy to learn useful invariants and then measure how much of that behavior survives the sim-to-real transition.

Evaluation is a separate phase, not if loss < x

Training does not mean a policy is safe to control hardware.

The repository explicitly separates the evaluation domain and includes several surfaces:

Local evaluation
Software-in-the-Loop (SiL)
Hardware-in-the-Loop (HiL)
OSMO-managed evaluation

The LeRobot documentation shows a particularly concrete example: an ACT policy controlling a UR10E through ROS 2.

The inference node consumes:

/joint_states
/camera/color/image_raw

and can publish commands on:

/lerobot/joint_commands

The loop uses a reference value of 30 Hz.

But the most important parameter is probably another one:

enable_control=false

The documentation recommends first running the policy without sending commands to the robot and observing predictions on /lerobot/status. Only afterward is real control enabled.

That separation between inferring and acting is an idea that belongs in any agentic system touching the physical world.

Before allowing irreversible action:

observe
→ predict
→ evaluate
→ authorize
→ act

Not:

model says yes
→ motor moves

ONNX and TensorRT: the trained model is still not the product

Once the policy has been validated, the next problem is running it efficiently enough at the edge.

The toolchain supports exporting and packaging models in formats such as ONNX and TensorRT.

That step is critical because training and inference live under different constraints.

During training we want flexibility:

PyTorch
large GPUs
batching
instrumentation
checkpoints

On the robot we want something closer to:

predictable latency
limited memory
controlled power consumption
stable runtime
hardware acceleration

So the full flow is really:

research model
      ↓
evaluated model
      ↓
optimized model
      ↓
versioned artifact
      ↓
container / edge runtime

The edge should not receive a checkpoint simply because it was the last one to finish training.

It should receive a promoted artifact after evaluation.

GitOps for robots: deployment needs rollback too

With one robot, updating a policy manually may be acceptable.

With twenty, one hundred, or robots spread across multiple plants, another problem appears: version skew.

Which robot is running which policy?

Which version should it run?

How do we roll back?

At T3 and T4, Microsoft introduces FluxCD and a GitOps model.

Git
 ↓ desired state
FluxCD
 ↓ reconcile
site / cluster
 ↓
robot runtime

Git becomes the declarative source of the desired version.

A rollback no longer means “SSH into the robot and remember which command we ran.”

It can become reverting declared state.

But the repository adds an important precision: before a policy changes on physical hardware there must be a deployment gate.

That distinguishes two ideas that are often mixed together:

fleet delivery is not the same as fleet intelligence.

T4 primarily addresses delivery, connectivity, identity, and gating across multiple sites.

T5 —still on the roadmap— aims to add drift detection, retraining, and fleet-wide analytics.

That distinction is healthy.

Distributing software to one hundred robots is already a hard problem.

Automatically closing the telemetry → drift → retraining → deploy loop is another, even more dangerous problem.

The Tier Model documentation itself warns against fully autonomous retraining from production data: a degraded distribution could end up reinforcing the exact behavior we want to correct.

Agents appear above the pipeline, not inside the control loop

Here the repository connects Physical AI with another trend: agentic engineering.

The README makes it clear that agents are optional. Stages can be executed manually through CLIs and APIs.

That matters.

Agentic AI is placed as an orchestration layer over existing deterministic systems:

human instruction
      ↓
agent
      ↓
selects pipeline
      ↓
CLI / APIs / workflows
      ↓
OSMO / Azure ML / storage / evaluation

It should not replace the verifiable mechanisms underneath.

The repository contains specific Copilot agents. One of the most interesting is OSMO Training Manager, designed to manage multi-turn conversations around imitation-learning jobs.

It can:

  • validate prerequisites;
  • prepare a run;
  • ask for confirmation before launching GPU workloads;
  • submit the job to OSMO;
  • follow logs;
  • analyze loss and checkpoints;
  • query Azure ML / MLflow metrics;
  • run post-training evaluation.

Its configuration includes explicit defaults for ACT, training steps, batch size, learning rate, and checkpoint frequency.

This illustrates an architecture we will probably see much more often:

probabilistic agent
      ↓
controls deterministic tools
      ↓
reproducible workflows
      ↓
versioned artifacts
      ↓
verifiable gates

The agent reduces operational complexity.

But the pipeline still exists if the agent is removed.

That design is far more robust than building the whole system as one giant conversation with an LLM.

Edge-first does not mean cloudless

Return to the TechRadar article.

It can be tempting to read “Physical AI needs edge” and conclude that cloud stops mattering.

The opposite is true.

Cloud remains extremely useful for:

  • storing large datasets;
  • training with multiple GPUs;
  • maintaining a model registry;
  • comparing experiments;
  • running massive simulation;
  • distributing artifacts;
  • observing multiple sites;
  • coordinating identity and access.

What changes is which responsibility can tolerate cloud distance.

We can view it like this:

EDGE — time critical
────────────────────────
sensors
state estimation
inference
local safety
control
actuators

CLOUD — elastic time
────────────────────────
datasets
simulation
training
experiment tracking
model registry
analytics
fleet delivery control-plane

A mature architecture does not choose between cloud and edge.

It decides what must keep working when the link between them disappears.

The repository also shows why “production-ready” needs nuance

Here it is worth separating the project’s ambition from its current state.

The README uses the term production-ready, but the documentation itself is more cautious.

T5 remains a roadmap item.

Several T3/T4 pieces are still presented as advanced recipes or evolving documentation.

More importantly, the project publishes a fairly candid STRIDE threat model.

That analysis identifies 19 threats, including critical and high-risk items that are still open.

Examples include:

  • Terraform state stored locally with plaintext secrets as a critical risk;
  • OSMO API authentication disabled in one configuration as a high risk;
  • a Model Encryption Key stored in a ConfigMap instead of a Secret as a high risk.

That does not invalidate the project.

Quite the opposite: publishing its own limits is a sign of maturity.

But it does mean “production-ready” should not be interpreted as:

clone the repo, run Terraform, and immediately connect a critical industrial arm.

It should be interpreted more like:

there is a serious architecture, executable code, automation, and an explicit path toward production, but every deployment still needs its own hardening, threat model, safety case, and validation.

In Physical AI, that distinction matters enormously.

The big idea: the product is the lifecycle, not the model

The most interesting lesson from the repository is not Azure Arc.

It is not OSMO.

It is not LeRobot.

It is not even Isaac Lab.

It is the way all those pieces connect.

CAPTURE
  ROS 2
    ↓
DATA
  MCAP / LeRobot
    ↓
CURATE
  validate / inspect
    ↓
LEARN
  RL / IL / VLA
    ↓
SIMULATE
  Isaac / Cosmos
    ↓
EVALUATE
  local / SiL / HiL
    ↓
OPTIMIZE
  ONNX / TensorRT
    ↓
REGISTER
  versioned artifact
    ↓
DELIVER
  GitOps / Flux / Arc
    ↓
RUN
  edge inference
    ↓
OBSERVE
  telemetry / outcomes

That cycle is the real Physical AI system.

The model is one replaceable piece inside the lifecycle.

And that connects with what is happening in software agents too: value is shifting from the isolated model toward the harness that turns it into an operational capability.

In Physical AI, we could express a similar idea:

Physical AI
=
Model
+ Sensors
+ Data Pipeline
+ Simulation
+ Training
+ Evaluation
+ Edge Runtime
+ Safety Gates
+ Deployment
+ Observability

Remove any one of those layers and a demo may still work.

Scaling without them is another story.

From AI that answers to AI that participates in the world

Chatbots made it natural to talk to a machine.

Agents are making it natural to delegate digital work.

Physical AI adds the next leap:

intelligence begins to participate directly in physical processes.

That raises the engineering bar.

A hallucination in a conversation can be annoying.

An error from a policy controlling hardware can have economic or physical consequences.

That is why the future of Physical AI will probably not be defined only by who has the best foundation model.

It will also be defined by who builds the best system for:

  • collecting experience;
  • turning it into trustworthy data;
  • training reproducibly;
  • simulating difficult cases;
  • evaluating before acting;
  • optimizing for edge;
  • deploying with rollback;
  • observing real behavior;
  • preventing a bad update from propagating.

Microsoft’s repository is interesting precisely because it tries to turn that list into software.

It is still evolving and several advanced layers are unfinished.

But it provides a concrete view of where Physical AI engineering may go:

not a robot connected to an API, but a complete learning, validation, and operations chain in which the edge acts and the cloud improves what will act next.

Sources