Written by Oscar Erbetta
If you're migrating or rehosting your Linux or GNU-like VMs from another cloud provider to AWS, you might notice that your instance starts but the network fails to initialise properly.
This usually happens because the source VM is running a cloud-specific customised kernel (like Azure's or Google Cloud's). These kernels are tailored to their original cloud environment and don't fully support AWS networking out of the box.
The fix?
You need to switch the Linux kernel to an AWS-compatible kernel before migrating. In this post, I'll guide you through how to do that, focusing on Ubuntu VMs as an example.
While many AWS MGN rehosting steps are covered elsewhere, I’ll also dive into troubleshooting some common networking problems that stem from kernel issues.
Why Switching the Kernel Matters
Cloud providers often customize Linux kernels to work best with their infrastructure (drivers, metadata servers, network interfaces).
When you move a VM to AWS without switching kernels:
The system can't correctly detect AWS's Elastic Network Interfaces (ENIs).
Cloud-init tries to apply the old cloud's networking settings.
Your instance boots but becomes unreachable over the network.
By switching to an AWS-supported kernel, you ensure your VM runs properly in the AWS environment, recognises networking interfaces, and can complete the migration successfully.
Migration Steps Overview
Here’s a high-level view of the migration and replication process with AWS MGN:
Limitations
Make sure you read the OS limitations when doing rehosts using AWS MGN here: https://docs.aws.amazon.com/mgn/latest/ug/Supported-Operating-Systems-Linux.html
Some cloud provider-specific kernels are not supported!
Prepare the Source VM
Check the running kernel by using the following command:uname -r
If the output shows a cloud-specific kernel (e.g., azure, google, etc.) at the end, you need to switch kernels.
Install AWS Replication Agent on Source VM:
Update the GCC compiler if the replication agent fails to install.
sudo apt-get update
sudo apt-get install gcc-12
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
sudo update-alternatives --config gcc
sudo apt-get install linux-headers-$(uname -r)
Use the command below to download the installer. This will fetch it from the AWS Sydney region. If you're in a different region, you can find the appropriate download URL: https://docs.aws.amazon.com/mgn/latest/ug/linux-agent.html
cd /tmp && wget -O ./aws-replication-installer-init https://aws-application-migration-service-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/latest/linux/aws-replication-installer-init
Switch to an AWS-Compatible Linux Kernel
Install the AWS-tuned kernel:sudo apt install linux-aws
Verify the kernel installation:dpkg --list | grep linux-image
Update the system to boot from the AWS kernel. (Do not reboot the machine):sudo vim /etc/default/grub
By default GRUB_DEFAULT value will be 0. Update it with the AWS kernel:
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.5.0-1024-aws" (update the version based on the aws kernel version installed in the source VM)
Do not reboot the VM at this point, as a reboot will apply the kernel changes on the source VM, which could cause the Azure VM to enter a generalised state.
Disable Cloud-Specific Services
Identify running services:sudo systemctl --type=service
Disable any unnecessary cloud-specific services (example for Azure services):
sudo systemctl disable hv-kvp-daemon.service
sudo systemctl disable walinuxagent.service
sudo systemctl disable walinux-agent-network-setup.service
Modify this step depending on the original cloud provider.
Reconfigure Cloud-Init Data Source
Change the cloud-init data source to AWS EC2:sudo dpkg-reconfigure cloud-init
Unselect the old cloud's datasource and select EC2.
Press Spacebar to select, Tab to move, and Enter to confirm.
Install the replication agent and start replication process
Add execute permission, run installer and follow the prompts:
sudo chmod +x aws-replication-installer-init
sudo ./aws-replication-installer-init
If you get this error:
Error: Kernel headers are missing or not properly configured. Please install the appropriate kernel headers. Installation failed.
See if you have enabled 'Secure Boot' on the Ubuntu VM. Disable it and reboot. It should let you install the replication agent after this.
Once replication is complete, you can perform a Cutover Instance when you're ready to migrate.
Troubleshooting
Instance not bringing network up
After launching your Test Instance or Cutover Instance in AWS MGN:
If the instance has a private IP assigned but does not respond to pings (despite correct Security Group rules), it’s likely:
Cloud-init is misconfigured.
The old cloud kernel is still running.
To fix:
Create a rescue instance (Using an Ubuntu AWS AMI) in the same AZ as the target machine.
Stop the target (replicated) instance and detach the root volume.
Attach the root volume as a secondary volume to the rescue instance.
Mount the root volume on the rescue instance.
To get the device name:
sudo lsblk
Mount the device:
sudo mount <device name> /mnt
Invoke a chroot environment:
sudo bash -c 'for m in dev proc run sys; do mount -o bind /$m /mnt/$m; done'
sudo chroot /mnt
Add the VPC DNS .2 IP resolver into /etc/resolv.conf:
sudo echo "nameserver <x.x.x.2 IP>" > /etc/resolv.conf
Just in case:
Re-install linux-aws kernelsudo apt update
sudo apt install linux-aws
Check and disable cloud-specific services (example of Azure services):
sudo systemctl disable hv-kvp-daemon.service
sudo systemctl disable walinuxagent.service
sudo systemctl disable walinux-agent-network-setup.service
Change the data source to AWS EC2
dpkg-reconfigure cloud-init
Rename cloud-specific config files. Sometimes these network files keep cloud-specific MAC Addresses and/or IPs so it’s better to have new network files (Azure example):
mv /etc/cloud/cloud.cfg.d/10-azure-kvp.cfg /etc/cloud/cloud.cfg.d/10-azure-kvp.cfg.disabled
mv /etc/cloud/cloud.cfg.d/90-azure.cfg /etc/cloud/cloud.cfg.d/90-azure.cfg.disabled
Update the network config 51-netcfg.yaml
sudo nano /etc/netplan/51-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
ens5:
dhcp4: true
Check
/etc/netplan/50-cloud-init.yaml
for any leftover MAC address hardcoding.Clean cloud-init logs:
sudo cloud-init clean --logs
Update the grub and initrd image
sudo update-grub && sudo update-grub2
sudo update-initramfs -u -v
Make sure you have deleted all cloud-specific modules (Azure example):
sudo apt purge linux*-azure linux-azure* walinuxagent
Clean up
/etc/fstab
(remove non-root mounts related to the old cloud).Exit chroot and unmount:
exit; umount -fl /mnt
Detach the volume, re-attach it as the root volume to the original instance, and boot it.
Reference: StratusGrid
Conclusion
Migrating Linux VMs from other clouds to AWS using AWS MGN can be straightforward, but cloud-init and cloud-specific kernels can cause networking issues after replication. If your instance boots but isn’t reachable, most likely the old cloud settings are still interfering.
Switching to the AWS-compatible kernel, disabling old services, and reconfiguring cloud-init usually resolves the problem. Hopefully, these steps help you avoid common pitfalls and get your environment running smoothly in AWS.