Switching Linux Kernels to Fix Network Issues During VM Rehosting to AWS

Written by Oscar Erbetta

If you're migrating or rehosting your Linux or GNU-like VMs from another cloud provider to AWS, you might notice that your instance starts but the network fails to initialise properly.

This usually happens because the source VM is running a cloud-specific customised kernel (like Azure's or Google Cloud's). These kernels are tailored to their original cloud environment and don't fully support AWS networking out of the box.

The fix?
You need to switch the Linux kernel to an AWS-compatible kernel before migrating. In this post, I'll guide you through how to do that, focusing on Ubuntu VMs as an example.

While many AWS MGN rehosting steps are covered elsewhere, I’ll also dive into troubleshooting some common networking problems that stem from kernel issues.

Why Switching the Kernel Matters

Cloud providers often customize Linux kernels to work best with their infrastructure (drivers, metadata servers, network interfaces).

When you move a VM to AWS without switching kernels:

  • The system can't correctly detect AWS's Elastic Network Interfaces (ENIs).

  • Cloud-init tries to apply the old cloud's networking settings.

  • Your instance boots but becomes unreachable over the network.

By switching to an AWS-supported kernel, you ensure your VM runs properly in the AWS environment, recognises networking interfaces, and can complete the migration successfully.

Migration Steps Overview

Here’s a high-level view of the migration and replication process with AWS MGN:

Limitations

Make sure you read the OS limitations when doing rehosts using AWS MGN here: https://docs.aws.amazon.com/mgn/latest/ug/Supported-Operating-Systems-Linux.html
Some cloud provider-specific kernels are not supported!

Prepare the Source VM

Check the running kernel by using the following command:
uname -r

If the output shows a cloud-specific kernel (e.g., azure, google, etc.) at the end, you need to switch kernels.

Install AWS Replication Agent on Source VM:

Update the GCC compiler if the replication agent fails to install.

sudo apt-get update
sudo apt-get install gcc-12
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
sudo update-alternatives --config gcc
sudo apt-get install linux-headers-$(uname -r)

Use the command below to download the installer. This will fetch it from the AWS Sydney region. If you're in a different region, you can find the appropriate download URL: https://docs.aws.amazon.com/mgn/latest/ug/linux-agent.html

cd /tmp && wget -O ./aws-replication-installer-init https://aws-application-migration-service-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/latest/linux/aws-replication-installer-init

Switch to an AWS-Compatible Linux Kernel

Install the AWS-tuned kernel:
sudo apt install linux-aws

Verify the kernel installation:
dpkg --list | grep linux-image

Update the system to boot from the AWS kernel. (Do not reboot the machine):
sudo vim /etc/default/grub

By default GRUB_DEFAULT value will be 0. Update it with the AWS kernel:

GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.5.0-1024-aws" (update the version based on the aws kernel version installed in the source VM)

Do not reboot the VM at this point, as a reboot will apply the kernel changes on the source VM, which could cause the Azure VM to enter a generalised state.

Disable Cloud-Specific Services

Identify running services:
sudo systemctl --type=service

Disable any unnecessary cloud-specific services (example for Azure services):

sudo systemctl disable hv-kvp-daemon.service
sudo systemctl disable walinuxagent.service
sudo systemctl disable walinux-agent-network-setup.service

Modify this step depending on the original cloud provider.

Reconfigure Cloud-Init Data Source

Change the cloud-init data source to AWS EC2:
sudo dpkg-reconfigure cloud-init

  • Unselect the old cloud's datasource and select EC2.

  • Press Spacebar to select, Tab to move, and Enter to confirm.


Install the replication agent and start replication process

Add execute permission, run installer and follow the prompts:

sudo chmod +x aws-replication-installer-init
sudo ./aws-replication-installer-init

If you get this error:
Error: Kernel headers are missing or not properly configured. Please install the appropriate kernel headers. Installation failed.
See if you have enabled 'Secure Boot' on the Ubuntu VM. Disable it and reboot. It should let you install the replication agent after this.

Once replication is complete, you can perform a Cutover Instance when you're ready to migrate.

Troubleshooting

Instance not bringing network up

After launching your Test Instance or Cutover Instance in AWS MGN:

If the instance has a private IP assigned but does not respond to pings (despite correct Security Group rules), it’s likely:

  • Cloud-init is misconfigured.

  • The old cloud kernel is still running.

To fix:

  • Create a rescue instance (Using an Ubuntu AWS AMI) in the same AZ as the target machine.

  • Stop the target (replicated) instance and detach the root volume.

  • Attach the root volume as a secondary volume to the rescue instance.

  • Mount the root volume on the rescue instance.

  • To get the device name:
    sudo lsblk

  • Mount the device:
    sudo mount <device name> /mnt

  • Invoke a chroot environment:
    sudo bash -c 'for m in dev proc run sys; do mount -o bind /$m /mnt/$m; done'
    sudo chroot /mnt

  • Add the VPC DNS .2 IP resolver into /etc/resolv.conf:
    sudo echo "nameserver <x.x.x.2 IP>" > /etc/resolv.conf

  • Just in case:
    Re-install linux-aws kernel
    sudo apt update
    sudo apt install linux-aws

  • Check and disable cloud-specific services (example of Azure services):
    sudo systemctl disable hv-kvp-daemon.service
    sudo systemctl disable walinuxagent.service
    sudo systemctl disable walinux-agent-network-setup.service

  • Change the data source to AWS EC2
    dpkg-reconfigure cloud-init

  • Rename cloud-specific config files. Sometimes these network files keep cloud-specific MAC Addresses and/or IPs so it’s better to have new network files (Azure example):
    mv /etc/cloud/cloud.cfg.d/10-azure-kvp.cfg /etc/cloud/cloud.cfg.d/10-azure-kvp.cfg.disabled
    mv /etc/cloud/cloud.cfg.d/90-azure.cfg /etc/cloud/cloud.cfg.d/90-azure.cfg.disabled

  • Update the network config 51-netcfg.yaml
    sudo nano /etc/netplan/51-netcfg.yaml 

network:
  version: 2
  renderer: networkd
  ethernets:
    ens5:
    dhcp4: true

  • Check /etc/netplan/50-cloud-init.yaml for any leftover MAC address hardcoding.

  • Clean cloud-init logs:
    sudo cloud-init clean --logs

  • Update the grub and initrd image
    sudo update-grub && sudo update-grub2
    sudo update-initramfs -u -v

  • Make sure you have deleted all cloud-specific modules (Azure example):
    sudo apt purge linux*-azure linux-azure* walinuxagent

  • Clean up /etc/fstab (remove non-root mounts related to the old cloud).

  • Exit chroot and unmount:
    exit; umount -fl /mnt

  • Detach the volume, re-attach it as the root volume to the original instance, and boot it.

Reference: StratusGrid

Conclusion

Migrating Linux VMs from other clouds to AWS using AWS MGN can be straightforward, but cloud-init and cloud-specific kernels can cause networking issues after replication. If your instance boots but isn’t reachable, most likely the old cloud settings are still interfering.

Switching to the AWS-compatible kernel, disabling old services, and reconfiguring cloud-init usually resolves the problem. Hopefully, these steps help you avoid common pitfalls and get your environment running smoothly in AWS.

AWS
boot
Cloud
kernel
linux
networking
05/02/2025