Migrate a Linux VPS to EC2 with AWS Application Migration Service
Overview
After my DNS migration article I was occupied with taking an exam for work, followed by the festive season.. nearly two months passed by until last week came when I finally managed to start migrating my servers to AWS. 96-fromsofia.net is actually hosted on S3+CloudFront, so there isn’t really a server there, but my git and mail servers were the targets that had to be migrated.
In this article, I will be going over the process of migrating a web server with static data from an external provider over to AWS using the Application Migration Service (AMS). I will also cover some of the caveats I encountered while migrating my mail server which can be seen as a dynamic application in the sense that data changes dynamically.
Prerequisites:
- Existing server running Linux, hosted with an external provider
- An AWS account
Step 1 - Set up AMS
Make sure you are logged into your AWS account in the region you want your server to be migrated in and head over to the EC2 console. Find the Launch instance button and next to it you should see a Migrate a server option.
Select it and you will be brought over to the AMS console. If this is the first time you are using AMS you will be asked to set up the migration hub.
Step 2 - Download and run the AMS agent
After the setup has finished, you will be able to select the Add server option and be taken to a new window that looks as such:
Choose if you want to replicate all disks or only certain ones. I personally chose all disks.
Next, you will need to provide the access and secret keys associated with an IAM user AMS will use to conduct the migration. If you don’t have one-click on the Create IAM user button and AWS will create it for you:
Note down the secret key ID as it is shown only once here at creation! If you don’t write it down you will need to create a new access and secret key ID.
Go back to the AMS console and enter the generated credentials.
You are now given two commands - one to download the agent and another one to run it. On CentOS/RHEL I had to install the elfutils-libelf-devel package for the AMS agent to work.
yum install -y elfutils elfutils-libelf-devel
On your external source server, run the first command to download the agent.
# wget -O ./aws-replication-installer-init.py https://aws-application-migration-service-eu-west-1.s3.eu-west-1.amazonaws.com/latest/linux/aws-replication-installer-init.py
--2023-01-03 16:41:24-- https://aws-application-migration-service-eu-west-1.s3.eu-west-1.amazonaws.com/latest/linux/aws-replication-installer-init.py
Resolving aws-application-migration-service-eu-west-1.s3.eu-west-1.amazonaws.com (aws-application-migration-service-eu-west-1.s3.eu-west-1.amazonaws.com)... 52.218.37.104, 52.218.112.56, 52.218.98.0, ...
Connecting to aws-application-migration-service-eu-west-1.s3.eu-west-1.amazonaws.com (aws-application-migration-service-eu-west-1.s3.eu-west-1.amazonaws.com)|52.218.37.104|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25289 (25K) [binary/octet-stream]
Saving to: ‘./aws-replication-installer-init.py’
./aws-replication-installer-init.py 100%[============================================================================>] 24.70K --.-KB/s in 0.02s
2023-01-03 16:41:24 (1.04 MB/s) - ‘./aws-replication-installer-init.py’ saved [25289/25289]
# ls aws-replication-installer-init.py
aws-replication-installer-init.py
#
Once the download has completed, paste the second command and wait until it has executed.
# python3 aws-replication-installer-init.py --region eu-west-1 --aws-access-key-id <ACCESS_KEY> --aws-secret-access-key <SECRET_KEY> --no-prompt
The installation of the AWS Replication Agent has started.
Identifying volumes for replication.
Identified volume for replication: /dev/vdb of size 1 GiB
Identified volume for replication: /dev/vda of size 25 GiB
All volumes for replication were successfully identified.
Downloading the AWS Replication Agent onto the source server... Finished.
Installing the AWS Replication Agent onto the source server... Finished.
Syncing the source server with the Application Migration Service Console... Finished.
The following is the source server ID: s-04c5bfc573a40abb4.
You now have 1 active source server out of a total quota of 20.
Learn more about increasing source servers limit at https://docs.aws.amazon.com/mgn/latest/ug/MGN-service-limits.html
The AWS Replication Agent was successfully installed.
#
During this process AMS will spin up an instance in the EC2 console which will be used to mount and synchronize your replicated storage on the testing instance you will be launching shortly. AMS will also be creating plenty of snapshots during the whole migration process. We will clean up those at the end of our migration.
Step 3 - Create AMI from snapshot
We will now need to create an AMI from one of the snapshots AMS created, this will be used as the image for our Test instance.
Go to the EC2 console and under Elastic Block Storage click on Snapshots.
Locate the latest snapshot for your source server.
From the Actions dropdown menu, select Create image from snapshot.
Specify a name for the image, review the default settings and if they comply with your source server’s architecture create the image.
Now from the EC2 console’s side panel under Images select AMIs
You should see your newly created AMI
Note if your source server has multiple hard disks, the process here differs.
- Create an image from the main hard disk’s snapshot
- Create a volume from each of the other required hard disk snapshots
- Later in Step 4 when editing your launch template:
- Select the AMI you created in 1.
- Under Storage add each of the volumes you created in 2. and attach them
Step 4 - Specify your launch settings
Upon successful completion, you should now see a server appear in the AMS console with the hostname of your external source server.
When the server first appears in AMS you will need to wait until the Initial Sync has finished. Depending on the server’s hard disks sizes, the time may differ.
Once the Initial Sync has finished, the instance should say Ready for Testing and have a Healthy status.
In the AMS console, click on the source server you’ve just migrated and select Launch settings from the banner.
Under General launch settings, select Edit and make any necessary adjustments.
I personally disable the right sizing option as I’ve noticed AWS will pick instances much larger than what I actually need. For example for my 1vCPU and 1GB RAM git server which sees about 5-15% utilization AWS suggested a c5.large instance which in my opinion was an overkill. Save your changes and you will be returned to the Launch settings view.
Below under EC2 Launch Template select the modify option, this will bring you to the EC2 Launch Template console where you can edit the default template.
Make sure you select the AMI you just created and then specify any additional settings for the EC2 instance such as your desired VPC, your security group auto-assign IP, any user-data (start-up scripts) you may want to run, etc.
As modifications to templates simply create a new version of the same template, you will also need to select the new version as the default one. Click on the Actions button and select Set default version.
Go back to the Launch settings screen in the AMS console and you should now see the EC2 instance type, subnet and all the other attributes from the newly updated Launch template.
Step 5 - Launch a testing instance
Now that you have configured your launch template, your source server has been added to AMS and is healthy, you can launch your test instance.
Head over to the source server in the AMS console, click on Test and Cutover and select Launch test instances.
AMS will now start the job and in the background you may see some EC2 instances being brought up and shut down in your console.
These are required for AMS to conduct the migration, you can click on the jobID to monitor the process.
The job log will look something similar to this and will keep on refreshing.
At the end it should say Job ended and no errors should be reported.
If you now go to the EC2 console, you should see an instance with the same name of your source server.
Step 6 - Testing and conducting the cutover
For my git server personally, the testing essentially looked something like this.
Confirm I can ssh to the new server’s IP with the old server’s ssh-key and username.
$ ssh -i <PATH_TO_KEY> -l username 52.214.1.168 -p <SSH_PORT>
The authenticity of host '[52.214.1.168]:<SSH_PORT> ([52.214.1.168]:<SSH_PORT>)' can't be established.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[52.214.1.168]:<SSH_PORT>' (ECDSA) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Tue Jan 3 16:39:12 2023 from <YOUR_IP>
$
Confirm the gitweb server is still running with terminal utilities such as dig and curl: dig a +short example.com ; curl -ILs http://example.com or use online tools such as:
https://mxtoolbox.com/SuperTool.aspx
https://www.webpagetest.org
Confirm I can still clone my repos with the same ssh-key, make commits and push them back to the repo.
git clone .. ; git add .. ; git commit .. ; git push ..
With these out of the way, we are now ready to launch our cutover instance. Go back to the AMS console and click on Test and cutover and select Launch cutover instances.
Similarly to the Test instance action, this will launch a job whose ID will appear in the AMS console.
Step 7 - Finalize cutover and clean-up
Now that the cutover job has completed and the instance has been launched, you can simply go through the same process of testing your application. Assuming you are happy with all the changes made, go back to the AMS console and under Test and cutover select the Finalize cutover option.:w
The Lifecycle view in the AMS console should now indicate that the Cutover is complete. Under actions, click on Mark as archived
At this point, go over to the Snapshots view in the EC2 console and select all snapshots you want to delete and remove them. I personally had no snapshots in this region and wanted to delete everything AMS related so I selected all of them when deleting.
AWS is good enough to not delete and inform you of any snapshots which are in use by either volumes or AMIs.
At the end, the only things left should be the snapshot behind your AMI and potentially snapshots related to additional volumes, as well as the now migrated EC2 instance.
Additional considerations
The above was an example of migrating a pretty simple web server where all essential data is static.
Many servers however run much more complex setups, where data is constantly being written, access to external services is needed and all sorts of other requirements that can make such migrations, well.. less pleasant. I personally had to migrate my mail server and I didn’t use AMS for the whole migration because it simply didn’t work.
Unfortunately, to test the mail server properly I had to ensure after launching the test instance I can send and receive mail. Then the same checks had to be verified for the cutover instance. Not sure if due to the static AMI I had selected, or due to the source server lagging, I ended up having data discrepancy between the source, testing and cutover servers.
To work around this I ensured dovecot, postifx, mariadb, spamd all were stopped before installing the AMS agent and while conducting the migration. Once the source server was generated in AMS, I simply created an AMI from the snapshots AMS took.
From this point onwards I didn’t use AMS and simply created an EC2 instance from my new AMI and made the necessary changes to the configuration files. Started the services I earlier stopped and amended my DNS records to point to the new EC2 instance. Receiving was working but sending wasn’t.
As I afterwards understood, AWS blocks outgoing port 25 traffic and encourages customers to use AWS SES. However, you are still free to submit a request to them and they will whitelist an elastic IP for you.
Additionally something else you should note is trying to stay away from static IPs in your source server’s configuration files.
If you use for example something like ListenAddress in your sshd_config set to a static IP value, then you may lock yourself out of the migrated server.
If this happens make sure you have selected at the very least a t3.X instance type so you can use the tty console from the EC2 dashboard.
The End
Well done for sticking around through this one. You should have now migrated your external Linux server from wherever it was running before over to AWS.
Assuming you’ve found AMS useful you should look into migrating Applications with it which is essentially migrating servers part of the same application together.
If you’ve enjoyed this, make sure to go ahead and look at the Articles section. My personal projects you can find on my git server. If you have a question or want to get in touch, feel free to email me.
Thank you for reading and have a good night!