Migrating Azure Application Insights URL Ping Tests to Standard Tests without losing alerts

Microsoft will retire classic Application Insights URL Ping Tests on September 30, 2026. For teams that still have this kind of availability monitoring configured, it is not enough to create a new Standard Test and assume Azure will take care of the rest.

In a recent real migration, we used Codex with GPT-5.6 Terra and reasoning effort medium, Azure CLI, Azure MCP, and Azure Resource Manager (ARM) to audit, migrate, and validate a set of production availability tests without interrupting existing monitoring.

The strategy was deliberately conservative:

discover first, reproduce the configuration, validate in parallel, move the alerts, and only then turn off the old tests.

That ordering turned out to be the most important part of the entire migration.

Microsoft confirms three especially relevant points:

  • URL Ping Tests retire on September 30, 2026.
  • Standard Tests incur charges when enabled.
  • Alert rules do not migrate automatically when the new test is created.

The problem

The environment contained several legacy Microsoft.Insights/webtests resources with:

properties.Kind = ping

An initial audit found:

7 classic URL Ping Tests
├── 4 enabled
└── 3 disabled

Azure Monitor rules associated with those tests also existed.

Instead of blindly migrating everything, the decision was to migrate only the four active tests.

That had two advantages:

  1. reduce risk;
  2. avoid creating unnecessary Standard Tests that would later generate executions and cost.

The session followed exactly that model: first discover the seven tests and their alerts, generate a migration report, and then limit the change to four active tests.

An important operational difference: Standard Tests do cost money

Classic URL Ping Tests were free. Standard Tests are billed per scheduled execution while enabled, so the migration also changes the economics of monitoring.

Cost grows mainly with three variables:

number of tests
×
number of locations
×
execution frequency

A useful approximation for monthly execution volume is:

runs/month ≈ tests × locations × (43,200 minutes / interval_in_minutes)

For example, four tests running from five locations every five minutes would produce approximately:

4 × 5 × (43,200 / 5)
= 172,800 scheduled executions / month

At a reference rate of US$0.0005 per execution, that would be on the order of:

172,800 × $0.0005 ≈ $86.40 / month

That figure is only an example for understanding scale. The applicable rate can vary by region, currency, and pricing changes, so it should always be checked on the official Azure Monitor pricing page before estimating a budget.

This cost must also be separated from the rest of Azure Monitor: alert rules and some notification mechanisms can be billed separately.

That economic difference reinforces two decisions that were already correct technically:

  • do not automatically migrate disabled legacy tests;
  • initially create Standard Tests with Enabled=false and enable them only after their configuration has been validated.

In other words, in an observability migration, cost is also part of the state we must preserve and control. It is not enough to ask “does it work the same?” We should also ask “how much will it cost to operate this configuration once it is active?”

1. The LLM agent and tools that executed the migration

The migration was orchestrated by Codex using GPT-5.6 Terra with reasoning effort medium.

The model’s role was not to replace Azure APIs or “guess” infrastructure state. Its job was to coordinate a verifiable procedure:

  1. translate the migration objective into queries and artifacts;
  2. discover the real Azure state through Azure MCP;
  3. cross-check findings with Azure CLI and ARM;
  4. build the ARM template and rollback plan;
  5. execute gradual changes only after validating the diff;
  6. query telemetry and alert state before each next step.

It is important to distinguish agent reasoning from the source of truth:

Codex / GPT-5.6 Terra (medium)
      ↓
planning, analysis, query and template generation
      ↓
Azure MCP + Azure CLI + ARM
      ↓
real inventory, state, deployment, telemetry, and alerts

The model variant describes the specific session that executed this migration, but reproducibility does not depend on the LLM. It depends on the commands, template, diffs, and Azure verifications documented below.

The operating principle was simple:

LLM proposes and coordinates
Azure exposes real state
ARM calculates and applies change
telemetry proves the outcome

2. Prepare Codex, Azure CLI, and Azure MCP

First, Azure CLI was authenticated and the active context was checked:

az login --tenant <tenant-id> --use-device-code

az account show \
  --query "{user:user.name, tenantId:tenantId, subscription:name, subscriptionId:id, state:state}" \
  --output json

Real identifiers were always treated as parameters:

Subscription: <subscription-id>
Tenant:       <tenant-id>
Resource RG:  <resource-group>

There is no need to include them in public documentation.

Azure MCP was added to Codex as a consolidated MCP server:

codex mcp add azure -- \
  npx -y @azure/mcp@latest server start --mode consolidated

codex mcp list

A useful property of this flow is that Azure MCP can rely on authentication already established through Azure CLI.

The separation of responsibilities became:

Codex / GPT-5.6 Terra
   │
   ├── Azure MCP ─────► discovery / Azure context
   │
   ├── Azure CLI ─────► checks / deployments / queries
   │
   └── ARM REST ──────► surgical resource updates

During discovery, Azure MCP was used in read-only mode.

3. Use Azure MCP to discover classic Ping Tests

The audit used Azure MCP’s arm router with these read-only subcommands:

arm.generate_query
arm.execute_query

First, arm.generate_query was asked to produce an Azure Resource Graph query to discover classic tests:

resources
| where type =~ 'microsoft.insights/webtests'
| where properties.Kind =~ 'ping'
| project id, name, resourceGroup, location, tags, kind, properties

It was then executed through arm.execute_query.

The query returned paginated results. Therefore, the first response was not assumed to be the complete inventory: the skipToken was checked and subsequent pages were consumed until the full resource set had been collected.

This detail matters in agentic automations. A partial result can look perfectly valid if the agent does not explicitly check whether a continuation exists.

Azure MCP provided discovery and resource context. Later changes were verified and executed deterministically with Azure CLI and Azure Resource Manager.

As an independent check, the Web Tests API was also queried directly through Azure CLI:

az rest --method get \
  --url "https://management.azure.com/subscriptions/<subscription-id>/providers/Microsoft.Insights/webtests?api-version=2022-06-15" \
  --output json

And existing metric alerts were inventoried:

az rest --method get \
  --url "https://management.azure.com/subscriptions/<subscription-id>/providers/Microsoft.Insights/metricAlerts?api-version=2018-03-01" \
  --output json

The pattern was deliberate:

MCP discovers
   ↓
CLI cross-checks
   ↓
ARM modifies only after dry run

4. Run a dry run before changing anything

An observability migration is especially dangerous because it can appear successful while silently breaking alerts.

That is why the first deliverable was not infrastructure.

It was a report:

URL_PING_MIGRATION_REPORT.md

For every test, the report inventoried:

- enabled/disabled state
- associated Application Insights resource
- endpoint
- HTTP method
- frequency
- timeout
- expected status code
- redirects
- retry
- content validation
- geographic locations
- associated alert rule

The goal was to build a model of:

CURRENT STATE
     │
     ▼
EQUIVALENT CONFIGURATION
     │
     ▼
PROPOSED CHANGE

before touching Azure.

5. Seek functional equivalence, not improvements

Standard Tests have capabilities the old Ping Tests did not have or did not expose in the same way:

  • TLS/SSL validation;
  • certificate-expiration validation;
  • custom headers;
  • different HTTP verbs;
  • request bodies;
  • additional validations.

It was tempting to enable these capabilities during the migration.

We did not.

The first objective was to achieve:

Legacy Ping Test ≈ Standard Test

while preserving existing semantics.

For the four tests, parameters were preserved such as:

HTTP method:              GET
Follow redirects:         true
Parse dependent requests: false
Retry:                    true
Expected status:          original
Frequency:                original
Timeout:                  original
Locations:                same locations

Initially, this was also preserved:

SSLCheck = false

Not because TLS verification is a bad idea, but because a migration is not the ideal moment to simultaneously change monitoring policy.

First parity.

Then improvements.

The ARM template preserved frequencies, timeouts, locations, expected codes, and content-validation rules from the original tests.

6. Never copy secrets from the old test into the repository

One availability test called an Azure Function using a URL similar to:

https://<function-app>/api/health?code=<function-key>

That code is a secret.

An automated migration can easily fall into this anti-pattern:

{
  "requestUrl": "https://example.azurewebsites.net/api/health?code=REAL_SECRET"
}

and end up storing the Function Key in Git.

The solution was to declare the template parameter as:

secureString

Conceptually:

{
  "healthcheckFunctionKey": {
    "type": "secureString"
  }
}

During deployment:

Old Ping Test
       │
       │ extract key
       ▼
     memory
       │
       │ secureString
       ▼
ARM deployment
       │
       ▼
Standard Test

The secret was never stored in:

Git
migration report
ARM template
documentation logs

In a real pipeline, it should be injected from a secret store or secure CI/CD mechanism, never from Git or documentation.

7. ARM What-If as a gate before deployment

Before applying the template, ARM What-If was executed.

The anonymized equivalent command was:

az deployment group what-if \
  --resource-group <resource-group> \
  --name <deployment-name>-whatif \
  --template-file URL_PING_STANDARD_TESTS.template.json \
  --parameters healthcheckFunctionKey="<secret-provided-at-runtime>" \
  --result-format ResourceIdOnly \
  --output json

The expected result was extremely specific:

+ 4 Microsoft.Insights/webtests

0 unexpected modifications
0 deletions

That condition acted as a gate.

If What-If had shown changes to Application Insights, alert rules, or other resources, deployment would have stopped.

This pattern is especially useful when an agent is generating infrastructure:

GENERATE
   ↓
WHAT-IF
   ↓
INSPECT
   ↓
APPLY

instead of:

GENERATE
   ↓
APPLY
   ↓
😬

The actual What-If confirmed exactly four new Microsoft.Insights/webtests resources and no changes to existing resources.

8. Create the Standard Tests disabled first

Deployment was executed using the ARM template:

az deployment group create \
  --resource-group <resource-group> \
  --name <deployment-name> \
  --template-file URL_PING_STANDARD_TESTS.template.json \
  --parameters healthcheckFunctionKey="<secret-provided-at-runtime>" \
  --query "{state:properties.provisioningState,timestamp:properties.timestamp}" \
  --output json

The new tests were not immediately created as active.

First:

Legacy Test       Enabled
Standard Test     Disabled

Then these were inspected:

Kind = standard
ProvisioningState = Succeeded
Frequency = expected
Timeout = expected
Locations = expected

Only afterward:

Standard Test → Enabled

For a temporary period, both resources existed.

That was intentional.

In addition to reducing technical risk, this pattern avoids generating executions —and therefore cost— while the resource is still being validated.

9. Validate telemetry before cutover

Once the Standard Tests were active, we had to prove they actually worked.

The Azure CLI Application Insights extension made it possible to query telemetry directly:

az extension add --name application-insights --yes --only-show-errors

az monitor app-insights query \
  --app <app-insights-name> \
  --resource-group <resource-group> \
  --analytics-query "
    availabilityResults
    | where timestamp > ago(90m)
    | where name endswith '-standard'
    | summarize
        runs=count(),
        successful=countif(success == '1'),
        failed=countif(success != '1'),
        latest=max(timestamp),
        maxDurationMs=max(duration)
      by name
    | order by name asc
  " \
  --output json

The first observed sample was approximately:

Test A     10 / 10 successful
Test B      9 / 9 successful
Test C      9 / 9 successful
Test D      9 / 9 successful

The legacy tests had not yet been modified.

The temporary architecture was:

             ┌── Legacy Ping Test ─────► endpoint
Application ─┤
             └── Standard Test ────────► endpoint

Both monitored the same application.

Running them in parallel made it possible to detect differences before the new test became the primary monitor.

10. The easy-to-forget detail: alerts

This is probably the most important lesson from the migration.

Creating:

foo-standard

does not magically make an alert that pointed to:

foo-legacy

start watching the new resource.

Microsoft explicitly warns that alert rules do not migrate automatically.

So cutover was explicit.

Each availability rule had to change from:

Alert Rule
    │
    ▼
Legacy Ping Test

to:

Alert Rule
    │
    ▼
Standard Test

The rules used contained at least two references that had to remain consistent:

properties.criteria.webTestId
properties.scopes

Updating only criteria.webTestId produced a scope-validation error. The fix was to also update the Web Test entry inside properties.scopes, while leaving the Application Insights component scope unchanged.

Operational characteristics were preserved as well:

Severity:           Sev1
Evaluation:         1 minute
Window:             5 minutes
Failed locations:   original threshold
Enabled:            true

The real migration updated both fields and preserved severity, window, evaluation frequency, and failed-location threshold.

11. ARM operations for surgical changes

Not every operation needed had a specific or convenient Azure CLI subcommand for this migration.

For those updates, a token was obtained with Azure CLI and ARM was called from PowerShell. That made it possible to preserve each resource’s complete document and modify only the required fields.

$token = az account get-access-token `
  --resource https://management.azure.com/ `
  --query accessToken -o tsv

$headers = @{ Authorization = "Bearer $token" }

Updating an alert required keeping two references consistent:

properties.criteria.webTestId
properties.scopes

Pseudocode for the change:

$alert = Get-ArmResource <metric-alert-resource-id>
$oldTestId = $alert.properties.criteria.webTestId
$newTestId = <standard-test-resource-id>

$alert.properties.criteria.webTestId = $newTestId
$alert.properties.scopes = $alert.properties.scopes | ForEach-Object {
  if ($_ -eq $oldTestId) { $newTestId } else { $_ }
}

Put-ArmResource <metric-alert-resource-id> $alert

The GET → modify the minimum → PUT pattern was reused to:

  • initially create Standard Tests with Enabled=false;
  • enable them after validating configuration;
  • disable the four legacy Ping Tests without deleting them;
  • associate <availability-notifications> with the four alerts;
  • execute and revert the controlled mutation test.

The value of the pattern is that it avoids manually reconstructing an entire resource from assumptions. We read the current state, modify only the intended field, and let ARM validate the resulting document again.

12. Connect the Action Group

The next component in the chain was the Action Group.

Conceptually:

Standard Test
     ↓
Alert Rule
     ↓
Action Group
     ↓
Email / Teams / Webhook / Function / etc.

This environment had an email Action Group.

Its configuration was inspected through ARM:

az rest --method get \
  --url "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Insights/actionGroups/<availability-notifications>?api-version=2023-01-01" \
  --output json

All real identifiers and recipients are omitted here:

Action Group: <availability-notifications>

Recipients:
  <operations-email-1>
  <operations-email-2>

After the migration, all four alerts were associated with the same Action Group.

That completed the path:

Synthetic request
      ↓
Standard Test
      ↓
Location failures
      ↓
Azure Monitor Alert
      ↓
Action Group
      ↓
Email

13. Disable, do not delete, the old tests

After verifying:

Standard Tests OK
        +
Alert Rules migrated
        +
Action Group connected

the classic Ping Tests were disabled.

They were not deleted.

Final state:

Legacy Ping Tests
Enabled = false

Standard Tests
Enabled = true

Why temporarily keep the old ones?

Rollback.

If a problem is discovered:

1. Enable legacy test
2. Restore alert webTestId
3. Restore alert scope

and the system can quickly return to the previous topology.

It is a classic deployment principle applied to observability:

disabling is reversible; deleting is not always reversible.

14. How do we prove the alert really works?

A Standard Test responding correctly proves only half the chain.

We still needed to verify:

Standard Test
     ↓
Alert Rule
     ↓
Action Group
     ↓
Email

Waiting for production to fail in order to test it was not attractive.

So we ran a controlled mutation test on the monitor.

The endpoint normally returned:

HTTP/1.1 200 OK

and the Standard Test expected:

ExpectedHttpStatusCode = 200

The test did not modify the application or endpoint. It temporarily changed the monitor’s expectation from 200 to 418:

$test.properties.ValidationRules.ExpectedHttpStatusCode = 418
Put-ArmResource <standard-test-resource-id> $test

Now this happened:

Application returns 200
        ↓
Monitor expects 418
        ↓
Synthetic test = FAILED

Only the monitor’s interpretation was temporarily altered.

15. Commands and queries used during the alert test

After changing the expectation, we verified that failures had accumulated from enough locations:

availabilityResults
| where timestamp > ago(15m)
| where name == '<standard-test-name>'
| summarize
    runs=count(),
    failed=countif(success != '1'),
    failedLocations=dcountif(location, success != '1'),
    latest=max(timestamp)

Availability agents ran the test again from five locations.

Result:

Location 1 → failure
Location 2 → failure
Location 3 → failure
Location 4 → failure
Location 5 → failure

The configured threshold required fewer failures than that to activate the rule.

To confirm the alert instance, Azure Alerts Management was queried:

GET https://management.azure.com/subscriptions/<subscription-id>/providers/Microsoft.AlertsManagement/alerts?alertRule=<metric-alert-resource-id>&timeRange=1h&includeContext=true&api-version=2019-03-01

The expected response for a successful test contains values such as:

monitorCondition = Fired
alertState       = New
severity         = Sev1

That is exactly what was observed, and the Action Group produced the real test emails.

In other words, validation was end-to-end:

HTTP endpoint
      ↓
Standard Test
      ↓
Synthetic failure
      ↓
Azure Monitor
      ↓
Alert rule
      ↓
Action Group
      ↓
Email inbox

As soon as failures and the alert were confirmed, the expectation was restored immediately:

$test.properties.ValidationRules.ExpectedHttpStatusCode = 200
Put-ArmResource <standard-test-resource-id> $test

The real test followed exactly this procedure:

200 → 418
     ↓
5 synthetic failures
     ↓
alert Fired / New / Sev1
     ↓
notification received
     ↓
418 → 200

The application and endpoint were never altered.

16. Why this kind of mutation test is so useful

A typical migration check might stop here:

✓ Resource exists
✓ provisioningState = Succeeded
✓ test returns green

But that does not prove you will know when it fails.

There are at least four independent systems:

1. Availability Test
2. Alert Rule
3. Alert Evaluation
4. Notification Channel

The mutation test validates all four.

It is the difference between testing:

"the monitor appears to be configured"

and testing:

"if this fails at 3 AM, somebody receives an alert"

17. The complete runbook

The procedure can be summarized as this pipeline:

1. Authenticate Azure CLI
      ↓
2. Start/configure Azure MCP in Codex
      ↓
3. Discover legacy tests via Resource Graph
      ↓
4. Consume pagination / skipToken
      ↓
5. Cross-check inventory with Azure CLI / ARM
      ↓
6. Generate migration report
      ↓
7. Select migration scope
      ↓
8. Estimate execution volume / operating cost
      ↓
9. Build equivalent Standard Test template
      ↓
10. Protect secrets with secureString
      ↓
11. ARM What-If
      ↓
12. Create Standard Tests disabled
      ↓
13. Validate configuration
      ↓
14. Enable Standard Tests
      ↓
15. Observe successful telemetry
      ↓
16. Redirect Alert Rules
      ↓
17. Validate Action Group
      ↓
18. Disable legacy tests
      ↓
19. Force controlled synthetic failure
      ↓
20. Confirm alert instance + notification
      ↓
21. Restore monitor
      ↓
22. Keep legacy tests temporarily for rollback

A useful way to think about the runbook is as a series of gates:

DISCOVERY COMPLETE?
      ↓ yes
DIFF EXPECTED?
      ↓ yes
STANDARD TESTS HEALTHY?
      ↓ yes
ALERT REFERENCES COHERENT?
      ↓ yes
NOTIFICATION VERIFIED?
      ↓ yes
CUTOVER COMPLETE

The agent did not advance simply because the previous command returned exit code zero. It advanced when Azure evidence satisfied the next gate.

18. What I would change for a large-scale migration

For four tests, individual inspection remains manageable.

With dozens or hundreds of availability tests, the procedure should become a declarative pipeline.

For example:

DISCOVER
    ↓
NORMALIZE
    ↓
DIFF
    ↓
ESTIMATE COST
    ↓
GENERATE
    ↓
WHAT-IF
    ↓
DEPLOY DISABLED
    ↓
VERIFY
    ↓
ENABLE
    ↓
OBSERVE
    ↓
CUTOVER ALERTS
    ↓
MUTATION TEST
    ↓
DEPRECATE LEGACY

And generate a manifest such as:

name: availability-api-prod

source:
  kind: ping

target:
  kind: standard

validation:
  expected_status: 200
  locations: 5
  retry: true

alert:
  severity: 1
  failed_locations: 2

rollback:
  legacy_test_retained: true

That would let an agent such as Codex reason over declarative state instead of improvising individual operations.

It would also make it possible to store, for every monitor:

source resource id
target resource id
normalized configuration
expected diff
estimated execution volume / cost
deployment result
telemetry evidence
alert migration evidence
mutation-test result
rollback instructions

The LLM can help coordinate the operation, while the system remains auditable without depending on its memory or hidden reasoning.

19. Six lessons from this migration

1. Creating the Standard Test does not finish the migration

The test is only one node in the monitoring system.

There are also:

alert rules
action groups
notification channels
rollback paths

2. Equivalence should come before improvements

Do not mix:

migration
+
new TLS policies
+
new timeouts
+
new endpoints
+
new alerts

in the same change.

First achieve functional equivalence.

Optimize afterward.

3. What-If should be mandatory

Especially when an agent is producing infrastructure.

A good agentic flow should be:

Agent proposes
Azure calculates diff
Human/agent evaluates
Azure applies

4. Secrets do not belong in the migration artifact

Even if the secret already exists inside the legacy resource, that does not make it acceptable to copy it into:

JSON
Markdown
logs
Git
prompt

Extract → use in memory → discard.

5. A monitor without a complete alert test is not fully validated

The correct final state was not only:

Availability = green

It was:

Availability = green
      +
Forced failure = alert
      +
Notification = received
      +
Configuration = restored

6. Cost is part of observability design too

When moving from free URL Ping Tests to Standard Tests billed per execution, seemingly technical decisions —such as five locations instead of one, or every five minutes instead of every fifteen— have a direct economic impact.

That is why a large-scale migration should explicitly include a cost diff alongside the configuration diff:

current configuration
      ↓
target configuration
      ↓
frequency × locations × tests
      ↓
estimated executions
      ↓
estimated operating cost

Optimization does not necessarily mean reducing coverage. It means making the trade-off between frequency, geographic redundancy, detection time, and cost explicit.

20. Official references

Conclusion

Migrating Application Insights URL Ping Tests to Standard Tests initially looks like a simple resource replacement.

In reality, it is a migration of an observability chain:

Endpoint
   ↓
Synthetic Test
   ↓
Availability telemetry
   ↓
Alert Rule
   ↓
Action Group
   ↓
Operator

If we migrate only the first block, we can end up with green dashboards and broken alerts.

The strategy that worked was much closer to a blue/green deployment:

Legacy ──────► active
Standard ────► build
Standard ────► validate
Alerts ──────► move
Standard ────► test end-to-end
Legacy ──────► disable

Codex with GPT-5.6 Terra (medium) was useful for planning, correlating information, and coordinating steps; Azure MCP provided discovery and context; Azure CLI and ARM provided deterministic and verifiable operations; and Azure telemetry and alerts supplied the evidence needed to continue or stop.

But the most important tool turned out to be the procedure:

inventory, dry run, preserve equivalence, estimate operating cost, validate in parallel, move alerts, test the failure path, and preserve rollback.

That pattern applies far beyond Application Insights.

It is, in essence, how any production observability migration should be done.