Rapidly uninstalling-reinstalling triggers can leave Astarte in a corrupt trigger state

[davidebriani]

We’ve observed that when Edgehog, an Astarte's API client, performs fast uninstall-and-install cycles for Astarte triggers, Astarte can end up in a corrupt state: triggers appear installed via APIs but do not function and do not deliver messages. This results in trigger-driven workflows, such as Edgehog's OTA campaigns, silently failing most of the time and only occasionally succeeding.

Investigation shows that Astarte’s trigger create and delete paths seem to be synchronous, so in principle they should be safe, yet quick successive operations still appear to lead to an inconsistent internal state.
One could also suspect that some Astarte's internal caches are not properly invalidated and refreshed with the updated trigger resources: e.g. the state in DUP processes.

The issue was found while using Edgehog v0.9.3 and Astarte v1.2-snapshot

Expected Behavior

• After trigger deletion and recreation, the system remains consistent and all installed triggers are usable. Event delivery is not impacted.

Actual Behavior

• Triggers appear as installed through APIs, but events are not delivered to the configured endpoint, Edgehog in this case.
• Upon deletion & re-creation of triggers, Trigger Engine emits warnings such as:

  |WARN| Trigger not found: "5a7d06ee-1d3d-420b-a1e0-72b053fbe612" function=retrieve_trigger_configuration/2
  |WARN| Error while processing event: {:error, :trigger_not_found} function=handle_simple_event/6

• State remains inconsistent until manual remediation steps are taken to clean the resources present in the database.

Impact

• Astarte clients that use triggers risk corrupting Astarte's state and breaking message delivery when changing trigger definitions (or trigger delivery policy definitions, possibly).

Open Questions

• Are there known race conditions in Trigger Engine or Realm Management when triggers are deleted and recreated in rapid succession?
• Could internal caches or propagation delays cause a “phantom trigger” state where API reports installed but retrieval resolves to not found?
• Would introducing stronger transactional guarantees, sequencing, or debounce on delete-create in the trigger path mitigate this?
• Are there recommended guardrails or best practices for clients performing trigger reconciliation to avoid corrupt states?

[davidebriani]

Steps to reproduce

Setup an Astarte instance with a realm, for example by following the Astarte in 5 minutes guide.
Here is the gist of it, in a bash script:

#!/usr/bin/env bash

echo "Initializing Astarte..."
git clone --depth=1 https://github.com/astarte-platform/astarte.git -b release-1.3
cd astarte
docker run --rm --user $(id -u):$(id -g) -v "$(pwd)/compose:/compose:z" astarte/docker-compose-initializer:1.2.0
docker compose pull
docker compose up -d

echo "Waiting for Astarte services to be ready..."
while true; do
    if curl -sf http://api.astarte.localhost/appengine/health >/dev/null && \
        curl -sf http://api.astarte.localhost/realmmanagement/health >/dev/null && \
        curl -sf http://api.astarte.localhost/pairing/health >/dev/null && \
        curl -sf http://api.astarte.localhost/housekeeping/health >/dev/null; then
        echo "Astarte services are ready."
        break
    fi
    echo "Still waiting for Astarte services..."
    sleep 3
done

echo "Creating Astarte realm..."
astartectl housekeeping realms create test --astarte-url http://api.astarte.localhost --realm-public-key test_public.pem -k compose/astarte-keys/housekeeping_private.pem -y

echo "Installing Astarte interfaces..."
astartectl realm-management interfaces sync standard-interfaces/*.json --astarte-url http://api.astarte.localhost -r test -k test_private.pem -y

Create a trigger, using an endpoint where you can receive and inspect the incoming trigger.

#!/usr/bin/env bash

export TOKEN=$(astartectl utils gen-jwt all-realm-apis -e 0 -k test_private.pem)

curl -X POST \
  http://api.astarte.localhost/realmmanagement/v1/test/triggers \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "data": {
      "name": "my_connection_trigger",
      "action": {
        "http_url": "<your-http-endpoint>",
        "http_method": "post"
      },
      "simple_triggers": [
        {
          "type": "device_trigger",
          "device_id": "*",
          "on": "device_connected"
        }
      ]
    }
  }'

Connect a device to Astarte, for example by using the container mentioned in the Astarte in 5 minutes guide:

#!/usr/bin/env bash

export DEVICE_ID=$(astartectl utils device-id generate-random)

docker run -it --net="host" -e "DEVICE_ID=$DEVICE_ID" -e "PAIRING_URL=http://api.astarte.localhost/pairing" -e "REALM=test" -e "PAIRING_JWT=$(astartectl utils gen-jwt pairing -k test_private.pem)" -e "IGNORE_SSL_ERRORS=true" astarte/astarte-stream-qt5-test:1.0.4

Once connected, DUP should contain a Data Updater process dedicated to the device and containing some internal caches, e.g. caching existing triggers.

Confirm that the existing trigger works by connecting and reconnecting the device: you should receive messages on your configured HTTP endpoint.

Now remove and install the trigger again.

#!/usr/bin/env bash

curl -X DELETE \
  -H "Authorization: Bearer $TOKEN" \
  http://api.astarte.localhost/realmmanagement/v1/test/triggers/my_connection_trigger

curl -X POST \
  http://api.astarte.localhost/realmmanagement/v1/test/triggers \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "data": {
      "name": "my_connection_trigger",
      "action": {
        "http_url": "<your-http-endpoint>",
        "http_method": "post"
      },
      "simple_triggers": [
        {
          "type": "device_trigger",
          "device_id": "*",
          "on": "device_connected"
        }
      ]
    }
  }'

Now if you disconnect and reconnect the device, the trigger doesn't appear to work anymore and your endpoint will not receive events.

You can inspect Trigger Engine's logs with:

docker logs astarte-astarte-trigger-engine-1 -f

And you should see it printing warning logs such as:

level=warning ts=2025-10-06T12:34:50.981+00:00 msg="Trigger not found: \"25b1bc39-f9a3-4fd7-812f-20b173951423\"" module=Astarte.TriggerEngine.EventsConsumer function=retrieve_trigger_configuration/2
level=warning ts=2025-10-06T12:34:50.982+00:00 msg="Error while processing event: {:error, :trigger_not_found}" module=Astarte.TriggerEngine.EventsConsumer function=handle_simple_event/6