PodVM-mkosi build fails to boot on AWS

[vlad-rusu]

Describe the bug

Building my images to run CAA in an AWS EKS environment, with PodVMs running on an m6a instance with SNP enabled, in eu-west-1 (where the prebuilt images are not available - nor can I copy an AMI as the snapshot isn't public..)

Initially I build from confidential-containers/cloud-api-adaptor/tree/main/src/cloud-api-adaptor/podvm -- setting CLOUD_PROVIDER=aws -- CAA spins up the podvm, the workload runs, but AA doesn't have awareness of my TEE env (SNP)

Then reading the docs I saw that confidential-containers/cloud-api-adaptor/tree/main/src/cloud-api-adaptor/podvm-mkosi is the right way to go and that I should use TEE_PLATFORM=amd (or snp)
Building this image works, but the CAA scheduled podvm never boots, no matter what I try

So from confidential-containers/cloud-api-adaptor/tree/main/src/cloud-api-adaptor/podvm-mkosi
TEE_PLATFORM=amd make debug (to get a debug image)
TEE_PLATFORM=amd make (regular image)

Uploaded raw using uplosi but also tried manually uploading, creating the snapshot and registering the ami - just in case uplosi would be the culprit.

None of these AMIs work, none of them boot (doesn't matter if I schedule a peer-pod or I try to create an EC2 instance from the AMI manually) - the EC2 instance stops itself consistently 12 seconds after kernel start, long before systemd reaches the point where process-user-data would run -- it stops at the point where kernel reaches the point where it should pivot to the root and start userspace.

Happy to provide full boot logs privately if needed, but the failure happens consistently at the same stage (≈12 seconds after kernel start) and always ends in Client.InstanceInitiatedShutdown.

Does anyone have any idea what is going on?

I am doing all this on a checked out version of the repo at the v0.16.0 tag

How to reproduce

confidential-containers/cloud-api-adaptor/tree/main/src/cloud-api-adaptor/podvm-mkosi
TEE_PLATFORM=amd make debug (to get a debug image)
TEE_PLATFORM=amd make (regular image)

Uploaded raw using uplosi but also tried manually uploading, creating the snapshot and registering the ami - just in case uplosi would be the culprit.

None of these AMIs work, none of them boot (doesn't matter if I schedule a peer-pod or I try to create an EC2 instance from the AMI manually) - the EC2 instance stops itself consistently 12 seconds after kernel start, long before systemd reaches the point where process-user-data would run -- it stops at the point where kernel reaches the point where it should pivot to the root and start userspace.

CoCo version information

v0.16.0

What TEE are you seeing the problem on

Snp

Failing command and relevant log output

[bpradipt]

Are you using uefi boot for the image?
Ref: https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/aws/raw-to-ami.sh#L162

[vlad-rusu]

Yes, UEFI boot configured on the AMI

[msalman-abid]

I'm having the same issue; I looked at the slack to figure out how to replace the deprecated 0.14.0 images from the public AMI catalog. I tried:

• the public reference image for 0.16.0 (ami-0f1c8092ca08f0c5c)
• building my own with TEE_PLATFORM=amd make (from the instructions here)

Both podvm images shut down on their own, and the cloud-api-adaptor fails with the following logs:

2025/11/24 02:12:02 [adaptor/cloud] created an instance podvm-nginx-1-65677598b8-b8tj5-7c5f731c for sandbox 7c5f731c69d1da31bbbaeab2d1e66abe16b3927472a180bfd92946c23d4cb81c
2025/11/24 02:12:02 [tunneler/vxlan] vxlan ppvxlan1 (remote 172.31.25.57:9000, id: 555001) created at /proc/1/task/10/ns/net
2025/11/24 02:12:02 [tunneler/vxlan] vxlan ppvxlan1 created at /proc/1/task/10/ns/net
2025/11/24 02:12:02 [tunneler/vxlan] vxlan ppvxlan1 is moved to /var/run/netns/cni-d5e5c7d3-15fd-667e-e94f-fbd6a210f6a7
2025/11/24 02:12:02 [tunneler/vxlan] Add tc redirect filters between eth0 and vxlan1 on pod network namespace /var/run/netns/cni-d5e5c7d3-15fd-667e-e94f-fbd6a210f6a7
2025/11/24 02:12:02 [adaptor/proxy] Listening on /run/peerpod/pods/7c5f731c69d1da31bbbaeab2d1e66abe16b3927472a180bfd92946c23d4cb81c/agent.ttrpc
2025/11/24 02:12:02 [adaptor/proxy] Trying to establish agent proxy connection to 172.31.25.57:15150
2025/11/24 02:14:18 [adaptor/proxy] Retrying failed agent proxy connection: dial tcp 172.31.25.57:15150: connect: connection timed out
2025/11/24 02:14:21 [adaptor/proxy] Retrying failed agent proxy connection: dial tcp 172.31.25.57:15150: connect: no route to host
2025/11/24 02:14:25 [adaptor/proxy] Retrying failed agent proxy connection: dial tcp 172.31.25.57:15150: connect: no route to host

[bpradipt]

@ldoktor @wainersm did you face this issue when working on the CI?

[wainersm]

@ldoktor @wainersm did you face this issue when working on the CI?

I didn't. In CI it's still using the packer built images. Let me try with mkosi one.

[vlad-rusu]

I am ok to use the packer image while mkosi is being looked into - but can I get TEE support from that image using AWS’s SNP? Based on docs I can’t or it is not clear how to build that image so that AA is complied for SNP.

Also not sure in using the packer image I can configure all guest components as required? AA and CDH specifically.

[msalman-abid]

I went back multiple versions {0.16.0, 0.14.0, 0.13.0} to find the last working version for the mkosi image; v0.13.0 looks like it works fine.

@vlad-rusu if you would like to do the same, you will have to checkout with git checkout v0.13.0 and build from there. There is an issue with gpg keys, so you will have to edit mkosi.conf to disable the check (example) and then build the image.

[bpradipt]

I'll take a look at this today..

[bpradipt]

The image does boot properly. I can boot it successfully as a standalone instance. However, during pod creation, the VM is being stopped after sometime. Some entity is shutting down the VM. I'm not sure what. The kubelet and containerd logs from the node are inconclusive. Continuing with debugging.

[vlad-rusu]

@bpradipt I can't boot the image standalone and also when using it with CAA (during pod creation) it is not CAA that stops it, it stops itself (because it doesn't boot). Takes about 12s for the VM to stop itself during boot, before loading/mapping any filesystem.

So which image have you tested please? Have you built a mkosi image from the latest version using the readme docs -- TEE_PLATFORM=amd make (from the instructions here)?

Also I'm not the only one seeing this behaviour..

[bpradipt]

Few additional updates.
When I mentioned that standalone instance creation worked fine, I didn’t enable SNP in cpu options. This proves why setting DISABLECVM: true in peer-pods-cm worked for me
I created a standalone instance with SNP enabled and it stopped after booting.
So it’s a combination of image and SNP enablement.

[bpradipt]

@vlad-rusu we are discussing this in the community slack channel in case you are interested to join - https://cloud-native.slack.com/archives/C04A2EJ70BX/p1764149041188289

[vlad-rusu]

Nice, I was in Slack but wasn't in the channel, now I am there.