Setting up SharePoint 2010 in Amazon Web Services

I was recently given the task to migrate a SharePoint environment to Amazon Web Services (AWS). With limited exposure to cloud computing, I was trying to figure out what shopping on Amazon had to do with moving a SharePoint environment. After a little bit of research, I learned about the cloud concept of an Infrastructure as a Service (IaaS) and that Amazon has been offering IaaS since 2006 with the introduction of Amazon Web Services.

Note to the Reader

The scenario that this blog post is based on required the complete migration of all servers out of a datacenter into the cloud with no on-premise domain controller (like many of you might encounter). The approach describe in t post might not be the best cloud solution for your situation, be sure to do your homework before choosing a path to the cloud. There are other IaaS providers; or perhaps a Platform as a Service (PaaS) such as Microsoft Azure is more suited for your situation.

What is Amazon Web Services?

Amazon Web Services (AWS) in the simplest of terms is a complete datacenter in the cloud. Traditionally, when a business starts, IT assets are purchased, installed, and maintained at a physical location (on-premise). The amount equipment required to set up a datacenter might include routers, switches, cabling, servers, cooling, power, battery backup, monitors, and much more. AWS can provide virtual equivalents of these assets for a relatively small price, reducing the total cost of ownership for the consumer. If the business or idea does not work out, simply shut down the devices and/or cancel your account.

Key Terms

While many of us are might be familiar with the hardware required to set up a

  • Elastic Computer Cloud (EC2) – EC2 is effectively your virtual server room, providing you access to other resources (ex. virtual machines, virtual private network, etc.)
  • Amazon Machine Image (AMI) – AMI is Amazon’s version of a virtual machine; you may be familiar with virtual machines if you have worked with VMware Workstation, Oracle VirtualBox, etc.
  • Virtual Private Cloud (VPC) – In the simplest of terms, a VPC is your virtual private network. If you have ever set up a home network, you probably know that all of the machines on your network have internal IP addresses (ex. 192.168.X.X or 10.0.X.X) and they all share a static IP to communicate with the rest of the Internet; this is your virtual private network.
  • Simple Storage Service (S3) – S3 is a scalable storage solution that allows you to store and retrieve any amount of data anytime.
  • Elastic Block Storage (EBS) – EBS allows you to create and store volumes (virtual hard drives) that can then be attached to AMI Instances.
  • Elastic IP – Is an IP address that can be allocated from Amazon at any time for a cost. An elastic IP can then be associated to any Instance of an AMI for public access.

Setting up a SharePoint Environment

Creating a VPC

If you plan on having more than one server (such as a 3-tier SharePoint environment), the first thing that you will want to do is set up a VPC; so that you can add your instances to it when you create them.

  1. Navigate to the VPC tab.
  2. Click Create a VPC
  3. Select the VPC that best fits your needs (VPC with a Single Public Subnet will be appropriate for most situations)
  4. Click Continue
  5. Edit your VPC IP CIDR Block and Public Subnet as needed (Defaults should be fine for most situations)
  6. Click Create VPC

Creating Security Groups

There are a variety of ways that security groups can be set up. What I ended up doing was creating one for each server type: Domain Controller (dc-sg), Database (db-sg), and SharePoint (sp-sg). For the dc-sg, db-sg, and sp-sg, I opened all incoming traffic for the private subnet (10.0.0.0/16) and port 3389 from anywhere (0.0.0.0/0). For sp-sg, I also opened incoming port 80 traffic from anywhere (0.0.0.0/0).

To create a security group:

  1. Navigate to the VPC tab
  2. Click Security Groups
  3. Click Create Security Group
  4. Enter Name and Description
  5. Select the VPC that you will adding your servers to
  6. Click Yes, Create

Instances

With your virtual network out of the way, it is time to stand up some servers. Depending on the type of environment that you are standing up, the type of instance and AMI used can vary as well as the price.

Choosing an Instance Type

At the time of this writing, AWS offers 4 standard instances that should be considered:

Name Memory CPU Storage
Small (m1.small) 1.7 GB 1 virtual core 160 GB
Medium (m1.medium) 3.75 GB 1 virtual core 410 GB
Large (m1.large) 7.5 GB 2 virtual cores 850 GB
Extra Large (m1.xlarge) 15 GB 4 virtual cores 1690 GB

View all Instance Types
View Instance Pricing

At the time of this writing, Microsoft minimum hardware requirements are as follows:

Tier Scenario Memory CPU Storage
Domain Controller All 8 GB 4-core 80 GB
Database Small deployment 8 GB 4-core 80 GB
Database Large deployment 16 GB 8-core 80 GB
SharePoint All 8 GB 4-core 80 GB

View Microsoft SharePoint 2010 minimum requirements

So what instance should you use? In my opinion, the answer is, like most answers in development, it depends. While AWS recommends Extra Large instances for all servers and High Memory Quadruple Extra Large (not listed) for the database server, this might necessarily be true for all situations. For example, if you were setting up a development environment, you could probably get away using all Small instances. If your SharePoint environment has relatively small usage, Large instances might work for you. Additionally, you need to consider licensing; for example, you could pay Amazon licensing for SQL Server by using one of their SQL AMIs or you could use a regular AMI and install SQL Server using your licensing. Take the time to know your environment; you might want to read the Amazon Web Services SharePoint whitepaper.

Creating Instances

Once you have decided the type of instance(s) that you plan to use:

  1. Navigate to the EC2 tab
  2. Click Instances
  3. Click Launch Instance
  4. Locate an AMI (ex. Microsoft Windows Server 2008 R2 Base) and click Select
  5. Select the Instance Type
  6. Select the VPC tab and select the VPC that you created previously
  7. Click continue
  8. (Optional) Select Prevent against accidental termination (this will prevent you from accidently deleting your instance)
  9. (Optional) Enter an IP Address (if you want your IP address to be in order like I do)
  10. Click Continue
  11. (Optional) Enter a Name (good way to know which server is which) (note: this is not the computer name, just a name within AWS)
  12. Click Continue
  13. Select or create a Key Pair (these are used to encrypt/decrypt the initial Windows administrator password) (make note of where you store your .pem file)
  14. Select the Security Group that you created previously or create one if needed
  15. Click Continue
  16. Check everything and then click Launch

Connecting to an Instance

At this point, an Instance is created from the AMI and is started. Monitor the State and Status Checks and wait for them to both to be green (running and 2/2 checks passed). This can take several minutes, so be patient. Once ready, we can give the instance an Elastic IP Address.

  1. Navigate to EC2
  2. Click Elastic IPs
  3. Click Allocate New Address
  4. Select VPC
  5. Click Yes, Allocate
  6. Check the box next to the Elastic IP that was just created
  7. Select Associate Address
  8. Select the desired instance
  9. Click Yes, Associate

Now that the instance has a public IP, we are ready to connect:

  1. Navigate to EC2
  2. Click Instances
  3. Check the box next to the desired instance
  4. Select the Instance Actions drop down list
  5. Select Get Windows Admin Password
  6. Open the .pem file that you downloaded when you created your key pair, copy ALL of its contents and paste it into the box labeled Private Key
  7. Click Decrypt Password
  8. Your password will be displayed (copy it, save it, memorize it….whichever you choose)
  9. Select the Instance Actions drop down list
  10. Click Connect
  11. From here, you can either download the RDP file or set one up manually and connect to your instance

Making Software Available

Now that you have your instances available, you might be wondering how you are going to install software such as SQL Server and SharePoint; after all, there is no virtual DVD drive. Like most things in the computer world, there are many ways to anything; I tend to stick to what I consider to be simple. I simply created a semi large volume, attached it to an instance, and copied/pasted in RDP. Alternatively, you could download it from MSDN from the server, setup Internet Explorer Enhanced Security (IEEC), and so on. Either way, I believe that it is a good idea to have this software on a volume somewhere that can be used by all machines (either by file share or dismount/mount). To create a volume and mount it to an instance:

  1. Navigate to EC2
  2. Click on Volumes
  3. Click Create Volume
  4. Enter the Size
  5. Select the Availability Zone that your instances reside in
  6. Do not select a Snapshot
  7. Click Yes, Create
  8. Check the box next to the volume that was just created (it is most likely the only one with a State of available)
  9. Select the …More drop down list
  10. Click Attach Volume
  11. Select the instance that you would live to attach it to
  12. Enter xvdf for the Device
  13. Click Yes, Attach
  14. Connect to the Instance
  15. In Windows, navigate to Administrative Tools > Computer Management > Disk Management
  16. Locate the volume that you just attached (it should be Offline)
  17. Right Click on the volume and select Online
  18. You should now be able to use the drive

References

Amazon Web Services, LLC. (2012). About AWS. Retrieved May 22, 2012, from http://aws.amazon.com/what-is-aws/.

Amazon Web Services, LLC. (2012). Amazon EC2 Instance Types. Retrieved May 23, 2012, from http://aws.amazon.com/ec2/instance-types/.

Cloudiquity. (2012). Difference between S3 and EBS. Retrieved May 23, 2012, from http://www.cloudiquity.com/2009/03/differences-between-s3-and-ebs/.

 

AWS Disaster Recovery Automation w/ Powershell

Amazon has provided several powerful SDKs for developers to interface with Amazon Web Services (AWS). Anything that can be completed using the AWS console can be accomplished via SDK. In this post, we will examine how .Net developers can leverage the AWS SDK to provide a simple disaster recovery plan for a 3-tier SharePoint environment (domain controller, database server, SharePoint server). This approach could be applied to nearly any environment; for more information, you might want to review the AWS Disaster Recovery Demo.

Setup Amazon Simple Email Service (SES)

The first thing you will want to do is apply for access to SES and setup a verified sender address; this may take several hours.

Get the AWS SDK

Download the AWS SDK for .NET.

Get the scripts

After the SDK has been installed, pick a place to store the scripts (ex. C:\AWS). Next, download the four files below (AWSConfig.ps1, AWSUtilities.ps1, DailySnapshots.ps1, and WeeklySnapshots.ps1) into your AWS directory. Alternatively, ff you hover over the code, the Script Highlighter Plugin will provide the ability to copy/paste, view source, etc. Additionally, create a directory called “Logs” inside of your AWS directory.

AWSConfig.ps1

[Download]

############## C O N F I G ##############

#AWS SDK Path 
Add-Type -Path "C:\Program Files (x86)\AWS SDK for .NET\bin\AWSSDK.dll"

#Access Keys
$accessKeyID="XXXXXXXXXXXXXXXXXXXX"
$secretAccessKey="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
$accountID = "############"

#Regions
#$serviceURL="https://ec2.us-west-1.amazonaws.com"
#$serviceURL="https://ec2.us-west-2.amazonaws.com"
#$serviceURL="https://ec2.us-east-1.amazonaws.com"

#Log
$LOG_PATH="C:\AWS\Logs\"

#Email
$FROM_ADDRESS = "you@example.com"
$ADMIN_ADDRESSES = "you@example.com","you2@example.com.com"

#Expiration
$EXPIRATION_DAYS = 5
$EXPIRATION_WEEKS = 4
$MAX_FUNCTION_RUNTIME = 60 # minutes

#Test
$TEST_URL = "http://example.com"

############## A W S  C L I E N T S ##############
#Global Amazon EC2 Client
$config=New-Object Amazon.EC2.AmazonEC2Config
$config.ServiceURL = $serviceURL
$EC2_CLIENT=[Amazon.AWSClientFactory]::CreateAmazonEC2Client($accessKeyID, $secretAccessKey, $config)

#Global Amazon SES Client
$SES_CLIENT=[Amazon.AWSClientFactory]::CreateAmazonSimpleEmailServiceClient($accessKeyID, $secretAccessKey)

 

AWSUtilities.ps1

[Download]

############## R E A D M E ##############
#--Variables in ALL CAPS live in AWSConfig.ps1

#Run next line only once; is is required to create source for Windows Event Log
#New-EventLog -Source "AWS PowerShell Utilities" -LogName "Application"

############## G L O B A L ##############
#global variable to hold email message
$global:email = ""

############## U T I L I T Y   F U N C T I O N S ##############

#Description: Returns true if function has been running longer than permitted 
#Returns: bool
function IsTimedOut([datetime] $start, [string] $functionName)
{    

    $current = new-timespan $start (get-date)

    If($current.Minutes -ge $MAX_FUNCTION_RUNTIME)
    {
        WriteToLogAndEmail "$FunctionName has taken longer than $MAX_FUNCTION_RUNTIME min. Aborting!"
        throw new-object System.Exception "$FunctionName has taken longer than $MAX_FUNCTION_RUNTIME min. Aborting!"
        return $true
    } 
    return $false
}
#Description: Adds a tag to an Amazon Web Services Resource
#Returns: n/a
function AddTagToResource([string] $resourceID, [string] $key, [string] $value)
{   
    try
    {
        $tag = new-object amazon.EC2.Model.Tag
        $tag.Key=$key
        $tag.Value=$value

        $createTagsRequest = new-object amazon.EC2.Model.CreateTagsRequest
        $createTagsResponse = $EC2_CLIENT.CreateTags($createTagsRequest.WithResourceId($resourceID).WithTag($tag))
        $createTagsResult = $createTagsResponse.CreateTagsResult; 
    }
    catch [Exception]
    {
        $function = "AddTagToResource"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }
}
#Description: Add carriage return characters for formatting purposes (ex. email)
#Returns: string[]
function FixNewLines([string[]] $text)
{    
    $returnText=""
    try
    {
        for($i=0;$i -le $text.Length;$i++)
        {
            $returnText+=$text[$i]+"`r`n"
        }
    }
    catch [Exception]
    {
        $function = "FixNewLines"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }

    return $returnText

}
#Description: Returns the current log name by determining the timestamp for the first day of the current week
#Returns: string
function GetLogDate
{
    $dayOfWeek = (get-date).DayOfWeek
    switch($dayOfWeek)
    {
        "Sunday" {$dayOfWeekNumber=0}
        "Monday" {$dayOfWeekNumber=1}
        "Tuesday" {$dayOfWeekNumber=2}
        "Wednesday" {$dayOfWeekNumber=3}
        "Thursday" {$dayOfWeekNumber=4}
        "Friday" {$dayOfWeekNumber=5}
        "Saturday" {$dayOfWeekNumber=6}
    }
    if($dayOfWeekNumber -eq 0)
    {
        $logDate = get-date -f yyyyMMdd
    }
    else
    {
        $logDate = get-date ((get-date).AddDays($dayOfWeekNumber * -1)) -f yyyyMMdd
    } 
    $logName = $logDate + ".txt"
    return  $logName  
}
#Description: Writes a message to a log file, console
#Returns: n/a
function WriteToLog([string[]] $text, [bool] $isException = $false)
{    
    try
    {
        if((Test-Path $LOG_PATH) -eq $false)
        {
            [IO.Directory]::CreateDirectory($LOG_PATH) 
        }
        $date = GetLogDate
        $logFilePath = $LOG_PATH + $date + ".txt"
        $currentDatetime = get-date -format G 
        add-content -Path $logFilePath -Value "$currentDatetime $text"
        write-host "$datetime $text"
        if($isException)
        {
            write-eventlog -Logname "Application" -EntryType "Information" -EventID "0" -Source "AWS PowerShell Utilities" -Message $text
        }
    }
    catch [Exception]
    {
        $function = "WriteToLog"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }    
}
#Description: Writes a email variable for later usage
#Returns: n/a
function WriteToEmail([string[]] $text, [bool] $excludeTimeStamp = $false)
{    
    try
    {
        if($excludeTimeStamp)
        {
            $global:email += "$text`r`n"
        }
        else
        {
            $datetime = get-date -format G 
            $global:email += "$datetime $text`r`n"
        }
    }
    catch [Exception]
    {
        $function = "WriteToEmail"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }
}

#Description: Write to log and email
#Returns: n/a
function WriteToLogAndEmail([string[]] $text, [bool] $isException = $false)
{
    WriteToLog $text $isException
    WriteToEmail $text
}
############## E M A I L   F U N C T I O N S ##############

#Description: Sends an email via Amazone Simple Email Service
#Returns: n/a
function SendSesEmail([string] $from, [string[]]$toList, [string]$subject, [string]$body)
{   
    try
    {
        # Make an Email Request
        $request = new-object -TypeName Amazon.SimpleEmail.Model.SendEmailRequest

        # "$request | gm" provides a lot of neat things you can do
        $request.Source = $from
        $list = new-object 'System.Collections.Generic.List[string]'
        foreach($address in $toList)
        {
            $list.Add($address)
        }
        $request.Destination = $list
        $subjectObject = new-object -TypeName Amazon.SimpleEmail.Model.Content
        $subjectObject.data = $subject
        $bodyContent = new-object -TypeName Amazon.SimpleEmail.Model.Content
        $bodyContent.data = $body
        $bodyObject = new-object -TypeName Amazon.SimpleEmail.Model.Body
        $bodyObject.text = $bodyContent
        $message = new-object -TypeName Amazon.SimpleEmail.Model.Message
        $message.Subject = $subjectObject
        $message.Body = $bodyObject
        $request.Message = $message

        # Send the message
        $response = $SES_CLIENT.SendEmail($request)
    }
    catch [Exception]
    {
        $function = "SendSesEmail"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }

}
#Description: Sends an status email to administrators
#Returns: n/a
function SendStatusEmail([string[]] $toAddress, [string] $successString = "", [string] $subject = "")
{ 

    try
    {   
        if($subject -eq "")
        {
            $subject = $successString + ": $ENVIRONMENT_NAME $ENVIRONMENT_TYPE $BACKUP_TYPE Backup"
        }

        $body = $global:email

        if($toAddress -eq $null -or $toAddress -eq "") { $toAddress = $ADMIN_ADDRESSES }

        SendSesEmail $FROM_ADDRESS $toAddress $subject $body
    }
    catch [Exception]
    {
        $function = "SendStatusEmail"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }
} 
############## I N S T A N C E   F U N C T I O N S ##############

#Description: Returns an Amazon Web Service Instance object for a given instance Id
#Returns: Instance
function GetInstance([string] $instanceID)
{
    try
    {
        $instancesRequest = new-object amazon.EC2.Model.DescribeInstancesRequest
        $instancesResponse = $EC2_CLIENT.DescribeInstances($instancesRequest.WithInstanceId($instanceID))
        $instancesResult = $instancesResponse.DescribeInstancesResult.Reservation
        return $instancesResult[0].RunningInstance[0]
    }
    catch [Exception]
    {
        $function = "GetInstance"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }
}
#Description: Returns all Amazon Web Service Instance objects
#Returns: ArrayList<Instance>
function GetAllInstances()
{
    try
    {
        $instancesRequest = new-object amazon.EC2.Model.DescribeInstancesRequest
        $instancesResponse = $EC2_CLIENT.DescribeInstances($instancesRequest)
        $instancesResult = $instancesResponse.DescribeInstancesResult.Reservation

         $allInstances = new-object System.Collections.ArrayList

        foreach($reservation in $instancesResult)
        {
            foreach($instance in $reservation.RunningInstance)
            {
                $allInstances.Add($instance) | out-null
            }
        }

        return $allInstances
    }
    catch [Exception]
    {
        $function = "GetAllInstances"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }
}
#Description: Returns an ArrayList of all running Amazon Web Service Instance objects that are
#Returns: Instance
function GetRunningInstances()
{
    try
    {

        $allInstances = GetAllInstances
        $runningInstances = new-object System.Collections.ArrayList

        foreach($instance in $allInstances)
        {
            if($instance.InstanceState.Name -eq "running")
            {
                $runningInstances.Add($instance) | out-null
            }
        }

        return $runningInstances
    }
    catch [Exception]
    {
        $function = "GetRunningInstances"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }
}
#Description: Gets the status of an Amazon Web Service Instance object for a given instance Id
#Returns: string
function GetInstanceStatus([string] $instanceID)
{
    try
    {
        $instance = GetInstance $instanceID
        return $instance.InstanceState.Name
    }
    catch [Exception]
    {
        $function = "GetInstanceStatus"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }
}
#Description: Gets the name of an Amazon Web Service Instance object for a given instance Id
#Returns: string
function GetInstanceName([string] $instanceID)
{
    try
    {
        $name = ""
        $instance = GetInstance $instanceID
        foreach($tag in $instance.Tag)
        {
            if($tag.Key -eq "Name")
            {
                $name = $tag.Value
            }
        }
        return $name
    }
    catch [Exception]
    {
        $function = "GetInstanceName"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }
}
#Description: Starts an Amazon Web Service Instance object for a given instance Id
#Returns: n/a
function StartInstance([string] $instanceID)
{    
    try
    {
        $instanceStatus = GetInstanceStatus $instanceID
        $name = GetInstanceName $instanceID
        if($instanceStatus -eq "running")
        {   
            WriteToLog "Instance $name ($instanceID) Already started"
            WriteToEmail "$name already started"
        }
        else
        {
            #Start instance    
            $startReq = new-object amazon.EC2.Model.StartInstancesRequest
            $startReq.InstanceId.Add($instanceID);    

            WriteToLog "Instance $name ($instanceID) Starting"    
            $startResponse = $EC2_CLIENT.StartInstances($startReq)
            $startResult = $startResponse.StartInstancesResult;

            #Wait for instance to finish starting. Unlike Stop instance,start one at a time (ex. DC, SQL, SP)
            $instancesRequest = new-object amazon.EC2.Model.DescribeInstancesRequest     

            $start = get-date

            do{
                #abort if infinite loop or otherwise
                if(IsTimedOut $start) { break } 

                start-sleep -s 5
                $instancesResponse = $EC2_CLIENT.DescribeInstances($instancesRequest.WithInstanceId($instanceID))
                $instancesResult = $instancesResponse.DescribeInstancesResult.Reservation
            }
            while($instancesResult[0].RunningInstance[0].InstanceState.Name -ne "running") 

            WriteToLog "Instance $name ($instanceID) Started"  
            WriteToEmail "$name started"               
        }
    }
    catch [Exception]
    {
        $function = "StartInstance"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }    
}
#Description: Starts one or more Amazon Web Service Instance object for a collection of instance Ids
#Returns: n/a
function StartInstances ([string[]] $instanceIDs)
{   
    try
    {
        $start = get-date

        foreach($instanceID in $instanceIDs)
        {
            StartInstance $instanceID            
        }

        $end = get-date
        $finish = new-timespan $start $end
        $finishMinutes = $finish.Minutes
        $finishSeconds = $finish.Seconds 
        WriteToLog "Start Instances completed in $finishMinutes min $finishSeconds sec"

    }
    catch [Exception]
    {
        $function = "Start Instances"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }

}
#Description: Starts all Amazon Web Service Instances
#Returns: n/a
function StartAllInstances()
{
    try
    {
        $instances = GetRunningInstances
        foreach($instance in $instances)
        {
            if($STARTALL_EXCEPTIONS -notcontains $instance.InstanceID)
            {
                StopInstance($instance)
            }
        }
    }
    catch [Exception]
    {
        $function = "StopRunningInstances"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }
}
#Description: Stops an Amazon Web Service Instance object for a given instance Id
#Returns: bool - is instance already stopped?
function StopInstance([string] $instanceID)
{    
    try
    {
        $instanceStatus = GetInstanceStatus $instanceID
        $name = GetInstanceName $instanceID
        if($instanceStatus -eq "stopped")
        {   
            WriteToLog "$name ($instanceID) Already Stopped"
            WriteToLog "$name already stopped"
            return $true
        }
        else
        {
            #Stop instance    
            $stopReq = new-object amazon.EC2.Model.StopInstancesRequest
            $stopReq.InstanceId.Add($instanceID);

            WriteToLog "Instance $name ($instanceID) Stopping"
            $stopResponse = $EC2_CLIENT.StopInstances($stopReq)
            $stopResult = $stopResponse.StopInstancesResult;  
            return $false      
        }
    }
    catch [Exception]
    {
        $function = "StopInstance"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }
}
#Description: Stops one or more Amazon Web Service Instance object for a collection of instance Ids
#Returns: n/a
function StopInstances([string[]] $instanceIDs)
{    
    try
    {    
        $statusInstanceIDs = new-object System.Collections.ArrayList($null)
        $statusInstanceIDs.AddRange($instanceIDs)

        #Stop all instances
        foreach($instanceID in $instanceIDs)
        {        
            if(StopInstance $instanceID)
            {
                $statusInstanceIDs.Remove($instanceID)
            }
        }
        #Wait for all instances to finish stopping
        $instancesRequest = new-object amazon.EC2.Model.DescribeInstancesRequest   

        $start = get-date        
        do
        {
            #abort if infinite loop or otherwise
            if(IsTimedOut $start) { break } 

            start-sleep -s 5
            foreach($instanceID in $statusInstanceIDs)
            {
                $status = GetInstanceStatus $instanceID
                if($status -eq "stopped")
                {
                    $name = GetInstanceName $instanceID
                    WriteToLog "Instance $name ($instanceID) Stopped"
                    WriteToEmail "$name stopped"
                    $statusInstanceIDs.Remove($instanceID)
                    break
                }
            }      
        }
        while($statusInstanceIDs.Count -ne 0)        

        $end = get-date
        $finish = new-timespan $start $end
        $finishMinutes = $finish.Minutes
        $finishSeconds = $finish.Seconds         
        WriteToLog "Stop Instances completed in $finishMinutes min $finishSeconds sec"
    }
    catch [Exception]
    {
        $function = "StopInstances"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }
}
#Description: Stops all Amazon Web Service Instances
#Returns: n/a
function StopAllInstances()
{
    try
    {
        [System.Collections.ArrayList]$instances = GetAllInstances
        foreach($instance in $instances)
        {
            if($STOPALL_EXCEPTIONS -notcontains $instance.InstanceID)
            {
                StopInstance($instance.InstanceID)
            }
        }
    }
    catch [Exception]
    {
        $function = "StopAllInstances"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }
}

############## S N A P S H O T   F U N C T I O N S ##############

#Description: Returns a Amazon Web Service Snapshot with a given snapshot Id
#Returns: Snapshot
function GetSnapshot([string] $snapshotID)
{
    try
    {
        $snapshotsRequest = new-object amazon.EC2.Model.DescribeSnapshotsRequest
        $snapshotsResponse = $EC2_CLIENT.DescribeSnapshots($snapshotsRequest.WithSnapshotId($snapshotID))
        $snapshotsResult = $snapshotsResponse.DescribeSnapshotsResult
        return $snapshotsResult.Snapshot[0]
    }
    catch [Exception]
    {
        $function = "GetSnapshot"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }
}
#Description: Returns all Amazon Web Service Snapshots
#Returns: Snapshot[]
function GetAllSnapshots
{
    try
    {
        $snapshotsRequest = new-object amazon.EC2.Model.DescribeSnapshotsRequest
        $snapshotsResponse = $EC2_CLIENT.DescribeSnapshots($snapshotsRequest.WithOwner($accountID))
        $snapshotsResult = $snapshotsResponse.DescribeSnapshotsResult
        return $snapshotsResult.Snapshot
    }
    catch [Exception]
    {
        $function = "GetAllSnapshots"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }
}
#Description: Returns the Description for Amazon Web Service Snapshot with a given snapshot Id
#Returns: string - description of snapshot
function GetSnapshotDescription([string] $snapshotID)
{
    try
    {
        $snapshot = GetSnapshot $snapshotID
        return $snapshot.Description
    }
    catch [Exception]
    {
        $function = "GetSnapshotDescription"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return $null
    }    
}
#Description: Deletes an Amazon Web Service Snapshot with a given snapshot Id
#Returns: n/a
function DeleteSnapshot([string] $snapshotID)
{    
    try
    {
        $name = GetSnapshotDescription $snapshotID                 
        WriteToLog "Snapshot $name ($snapshotID) Deleting"

        $deleteSnapshotRequest = new-object amazon.EC2.Model.DeleteSnapshotRequest
        $deleteSnapshotResponse = $EC2_CLIENT.DeleteSnapshot($deleteSnapshotRequest.WithSnapshotId($snapshotID))
        $deleteSnapshotResult = $deleteSnapshotResponse.DeleteSnapshotResult; 

        WriteToLog "Snapshot $name ($snapshotID) Deleted" 
        WriteToEmail "Snapshot Deleted: $name"    

    }
    catch [Exception]
    {
        $function = "DeleteSnapshot"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
    }

}

#Description: Creates an Amazon Web Service Snapshot for a given instance Id
#Returns: string - newly created snapshotID
function CreateSnapshotForInstance([string] $volumeID, [string] $instanceID)
{    
    try
    {
        #Generate meaningful description for snapshot
        $date = get-date -format yyyyMMddhhmmss
        $name = GetInstanceName $instanceID
        $description = "{0} {1} {2}" -f $name, $BACKUP_TYPE, $date

        WriteToLog "Instance $name ($instanceID) Creating Snapshot"

        $createSnapshotRequest = new-object amazon.EC2.Model.CreateSnapshotRequest
        $createSnapshotResponse = $EC2_CLIENT.CreateSnapshot($createSnapshotRequest.WithVolumeId($volumeID).WithDescription($description))
        $createSnapshotResult = $createSnapshotResponse.CreateSnapshotResult; 

        WriteToLog "Snapshot $description Created for $name ($instanceID)"
        WriteToEmail "$name snapshot successful"
        return $createSnapshotResult.Snapshot.SnapshotId
    }
    catch [Exception]
    {
        $function = "CreateSnapshotForInstance"
        $exception = $_.Exception.ToString()
        WriteToEmail "$name snapshot failed, Exception:"
        WriteToLogAndEmail "function: $exception" -isException $true
        return $null
    }
}
#Description: Creates Amazon Web Service Snapshots for a collection of instance Ids
#Parameters: $instanceIDs string[]
#Returns: n/a
function CreateSnapshotsForInstances([string[]] $instanceIDs)
{
    try
    {
        if($InstanceIDs -ne $null)
        {
            $volumesRequest = new-object amazon.EC2.Model.DescribeVolumesRequest
            $volumesResponse = $EC2_CLIENT.DescribeVolumes($volumesRequest)
            $volumesResult = $volumesResponse.DescribeVolumesResult
            foreach($volume in $volumesResult.Volume)
            {

                if($InstanceIDs -contains $volume.Attachment[0].InstanceId)
                {            
                    #Create the snapshot
                    $snapshotId = CreateSnapshotForInstance $volume.VolumeId $volume.Attachment[0].InstanceId           

                    #Wait for snapshot creation to complete
                    $snapshotsRequest = new-object amazon.EC2.Model.DescribeSnapshotsRequest   

                    $start = get-date             
                    do
                    {
                        #abort if infinite loop or otherwise
                        if(IsTimedOut $Start) { break } 

                        start-sleep -s 5
                        $snapshotsResponse = $EC2_CLIENT.DescribeSnapshots($snapshotsRequest.WithSnapshotId($snapshotId))
                        $snapshotsResult = $snapshotsResponse.DescribeSnapshotsResult
                    }
                    while($snapshotsResult.Snapshot[0].Status -ne "completed")            

                }            
            }  
        }
        else
        {
            WriteToLogAndEmail "Backup failed; no InstanceIDs to process"
        }
    }
    catch [Exception]
    {
        $function = "CreateSnapshotForInstances"
        $exception = $_.Exception.ToString()
        WriteToLogAndEmail "function: $exception" -isException $true
    }
}
#Description: Returns true if passed date is before the current date minus $EXPIRATION_DAYS value
#Returns: bool
function IsDailySnapshotExpired([datetime] $backupDate)
{
    try
    {
        $expireDate = (get-date).AddDays($EXPIRATION_DAYS*-1)
        return ($backupDate) -lt ($expireDate)
    }
    catch [Exception]
    {
        $function = "IsDailySnapshotExpired"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return false
    }    
}

#Description: Returns true if passed date is before the current date minus $EXPIRATION_WEEKS value
#Parameters: $backupDate datetime
#Returns: bool
function IsWeeklySnapshotExpired([datetime] $backupDate)
{
    try
    {
        $expireDate = (get-date).AddDays(($EXPIRATION_WEEKS * 7) * -1)
        return ($backupDate) -lt ($expireDate)
    }
    catch [Exception]
    {
        $function = "IsWeeklySnapshotExpired"
        $exception = $_.Exception.ToString()
        WriteToLog "function: $exception" -isException $true
        return false
    }    
}
#Description: Deleted old daily snapshots
#Parameters: n/a
#Returns: n/a
function CleanupDailySnapshots
{
    try
    {
        WriteToLog "Cleaning up daily snapshots"
        $deleteCount = 0

        $snapshots = GetAllSnapshots
        foreach($snapshot in $snapshots)
        {
            $description = $snapshot.Description
            $snapshotID = $snapshot.SnapshotId
            if($snapshot.Description.Contains("Daily"))
            {
                $backupDateTime = get-date $snapshot.StartTime
                $expired = IsDailySnapshotExpired $backupDateTime
                if($expired)
                {
                    DeleteSnapshot $snapshot.SnapshotId
                    $deleteCount ++
                    WriteToLog "$description ($snapshotID) Expired"
                }

            }
        }
        WriteToLogAndEmail "$deleteCount daily snapshots deleted"
    }
    catch [Exception]
    {
        $function = "CleanupWeeklySnapshots"
        $exception = $_.Exception.ToString()
        WriteToLogAndEmail "function: $exception" -isException $true
        return false
    } 
}
#Description: Deleted old weekly snapshots
#Parameters: n/a
#Returns: n/a
function CleanupWeeklySnapshots
{
    try
    {
        WriteToLog "Cleaning up weekly snapshots"
        $deleteCount = 0

        $snapshots = GetAllSnapshots
        foreach($snapshot in $snapshots)
        {
            $description = $snapshot.Description
            $snapshotID = $snapshot.SnapshotId
            if($snapshot.Description.Contains("Weekly"))
            {
                $backupDateTime = get-date $snapshot.StartTime
                $expired = IsWeeklySnapshotExpired $backupDateTime
                if($expired)
                {
                    DeleteSnapshot $snapshot.SnapshotId
                    $deleteCount ++
                    WriteToLog "$description ($snapshotID) Expired"
                }

            }
        }
        WriteToLogAndEmail "$deleteCount weekly snapshots deleted"
    }
    catch [Exception]
    {
        $function = "CleanupDailySnapshots"
        $exception = $_.Exception.ToString()
        WriteToLogAndEmail "function: $exception" -isException $true
        return false
    } 
}

 

DailySnapshot.ps1

[Download]


############## C O N F I G ##############
."C:\AWS\AWSConfig.ps1"

#Environment
$ENVIRONMENT_NAME = "My Environment"
$ENVIRONMENT_TYPE = "Development"
$BACKUP_TYPE = "Daily"
$stagingInstanceIDs="i-xxxxxxxx","i-xxxxxxxx","i-xxxxxxxx"

############## F U N C T I O N S ##############
."C:\AWS\AWSUtilities.ps1"

############## M A I N ##############

try
{
    $start = Get-Date
    WriteToLogAndEmail "$ENVIRONMENT_NAME $ENVIRONMENT_TYPE $BACKUP_TYPE Backup Starting" -excludeTimeStamp $true

    CreateSnapshotsForInstances $stagingInstanceIDs

    CleanupDailySnapshots

    WriteToLogAndEmail "$ENVIRONMENT_NAME $ENVIRONMENT_TYPE $BACK_UPTYPE Backup Complete" -excludeTimeStamp $true   

    $end = Get-Date
    $timespan = New-TimeSpan $start $end
    $hours=$timespan.Hours
    $minutes=$timespan.Minutes    
    WriteToEmail "Backup took $hours hr(s) and $minutes min(s)"

    WriteToEmail "Click here to test: $TEST_URL" -excludeTimeStamp $true
    SendStatusEmail -successString "SUCCESS"
}
catch
{
    SendStatusEmail -successString "FAILED"
}

 

WeeklySnapshot.ps1

[Download]

############## C O N F I G ##############
."C:\AWS\AWSConfig.ps1"

#Environment
$ENVIRONMENT_NAME = "My Environment"
$ENVIRONMENT_TYPE = "Development"
$BACKUP_TYPE = "Weekly"
$stagingInstanceIDs="i-xxxxxxxx","i-xxxxxxxx","i-xxxxxxxx"

############## F U N C T I O N S ##############
."C:\AWS\AWSUtilities.ps1"

############## M A I N ##############

try
{
    $start = Get-Date
    WriteToLogAndEmail "$ENVIRONMENT_NAME $ENVIRONMENT_TYPE $BACKUP_TYPE Backup Starting" -excludeTimeStamp $true

    StopInstances $stagingInstanceIDs
    CreateSnapshotsForInstances $stagingInstanceIDs
    StartInstances $stagingInstanceIDs

    CleanupWeeklySnapshots

    WriteToLogAndEmail "$ENVIRONMENT_NAME $ENVIRONMENT_TYPE $BACK_UPTYPE Backup Complete" -excludeTimeStamp $true

    $end = Get-Date
    $timespan = New-TimeSpan $start $end
    $hours=$timespan.Hours
    $minutes=$timespan.Minutes    
    WriteToEmail "Backup took $hours hours and $minutes to complete"

    WriteToEmail "Click here to test: $TEST_URL" -excludeTimeStamp $true

    SendStatusEmail -successString "SUCCESS"
}
catch
{
    SendStatusEmail -successString "FAILED"
}

 

Configuration

Configure AWSConfig.ps1

Open AWSConfig.ps1:

  1. AWS Access Path
  2. Provide the Access Key ID,  Secret Access Key, and Account ID for your account. Get Account ID, Access Key and Secret Access Key from AWS: My Account / Console > Security Credentials
  3. Uncomment the the region that your instances are running in
  4. Choose a location to store logs
  5. Provide a from address (must be verified in Amazon Simple Email Services (SES))

Configure AWSUtilities.ps1

Open AWSUtilities.ps1 and execute the line:

New-EventLog -Source "AWS PowerShell Utilities" -LogName "Application

. This will allow the scripts to add entries into the event log.

Configure DailySnapshots.ps1 & WeeklySnapshots.ps1

Open both DailySnapshots.ps1 and WeeklySnapshots.ps1 and:

  1. Verify the paths to AWSConfig.ps1 and AWSUtilities.ps1
  2. Set the ENVIRONMENT_NAME to whatever you want to call the environment (ex. the name of the customer). This variable is used in notification emails.
  3. Set the ENVIRONMENT_TYPE to the name of the environment (ex. production, development, etc.)
  4. BACKUP_TYPE should be set to the name of the backup (i.e. daily or weekly); however, you can customize if you like.
  5. Set stagingInstanceIDs to the instances that make up your environment

Test it out

When you execute the DailySnapshot.ps1 script, the script will create a snapshot of each volume for each instance that you provided without shutting them down. Once it is complete, you should receive an email stating the status of the backup. Similarly, when you execute WeeklySnapshot.ps1, the script will shut down the instances, create snapshots of each volume on each instance, start them back up in the order listed, and then send you an email notification. You can see this in action if you review the AWS Disaster Recovery Demo.

Increase disk size for an EC2 instance in AWS

While using Amazon Web Services (AWS) you may find that, when using one of the Amazon Machine Images (AMIs) provided, you may run out of disk space. There does not appear to be any way to “resize” an Elastic Block Store (EBS) volume; however, you can create a new one based on an existing snapshot and replace the current drive with a larger one. I have provided the instructions for doing so below.

  1. Log into the AWS Management Console
  2. Make sure that the Instance that you wish to change is not currently running
  3. Navigate to Elastic Block Store > Volumes
  4. Check the box next to the Volume that needs more space
  5. Click the More… drop down list and select Create Snapshot
  6. Enter a Name and a Description
  7. Navigate to Elastic Block Store> Snapshots
  8. Monitor the progress of the Snapshot for completion
  9. Once complete, navigate back to Elastic Block Store > Volumes
  10. Click Create Volume
  11. Enter the desired size for the new volume
  12. Make sure that you select the same Availability Zone for your instance
  13. Select the Snapshot that you just created
  14. Click Yes, Create
  15. Monitor the progress of the Volume for completion
  16. Once complete, Check the box next to the Volume that is currently connected to the Instance
  17. Make sure that no other Volumes are selected
  18. Click the More… drop down list and select Click the More… drop down list and select Detach Volume
  19. Uncheck the box selected and check the box next to the new Volume
  20. Click the More… drop down list and select Click the More… drop down list and select Attach Volume
  21. Select the Instance that you wish to change
  22. Change the value in Device from xvdf  to /dev/sda1
  23. Click Yes, Attach
  24. Navigate to Instances and start your Instance
  25. Once you are satisfied that the volume works, feel free to delete the Snapshot and old Volume if you like

You will need to take additional steps to extend the size of your partition to use the rest of the allocated space. The process for doing so will vary depending on the operating system that you are using.

Migrating a VMware VM to AWS

This document describes the steps necessary to migrate a VMWare virtual machine (.vmx and accompanying .vmdk) to the Amazon Web Services (AWS) cloud.

Setting up AWS Command Line Tools

There are many actions in AWS that can only be completed by using the AWS command line tools. This section describes the steps necessary to set up these tools (Evens, 2009).

Install Java

Download and install Java if you do not already have it: http://www.java.com/en/download/manual.jsp#win.

Download APIs

  1. EC2 Command line tools: http://aws.amazon.com/developertools/351
  2. Elastic Load Balancing tools: http://aws.amazon.com/developertools/2536?_encoding=UTF8&jiveRedirect=1

It is a good idea to pick somewhere to place all of your command line related items; for example: C:\AWS. Unzip both archives into this folder.

Get your AWS Certificates

  1. Go to https://aws-portal.amazon.com/gp/aws/securityCredentials
  2. Log in
  3. Scroll down to the X.509 section
  4. Create a new certificate (if one does not exist)
  5. Download the Private Key File (pk-.pem) and store in AWS directory
  6. Download the Certificate File (cert-.pem) and store in AWS directory
  7. KEEP THESE FILES PROTECTED – they provide access to your AWS account

Configure Environment Variables

Create a batch file in your AWS directory awsTools.bat and add the following text:

SET JAVA_HOME="C:\Program Files (x86)\java\jre6"
SET EC2_PRIVATE_KEY=c:\aws\aws-pk.pem
SET EC2_CERT=c:\aws\aws-cer.pem
SET EC2_HOME=c:\aws\ec2\ec2-api-tools-1.5.2.4
SET PATH=%PATH%;%EC2_HOME%\bin
SET AWS_ELB_HOME=D:\aws\ElasticLoadBalancing-1.0.15.1
SET PATH=%PATH%;%AWS_ELB_HOME%\bin
CLS
CMD

JAVA_HOME will need to be set to the appropriate path for your machine. Find java.exe (it will be a in a folder called bin) and set JAVA_HOME to the parent directory of bin. For example with java.exe at “C:\Program Files (x86)\java\jre6\bin\java.exe” set JAVA_HOME to “C:\Program Files (x86)\java\jre6”.

EC2_HOME and AWS_ELB_HOME both point to the folders you unzipped the APIs into.

Test Command

If you run awsTools.bat you should have a command prompt that you can run the EC2 tools from.
Type ec2-describe-regions at the command prompt
Your results should look something like the following:

REGION eu-west-1 eu-west-1.ec2.amazonaws.com
REGION us-east-1 us-east-1.ec2.amazonaws.com

If you get an error or nothing is returned, check your configuration.

Setting Up Your VMware Image

There are several changes that should be made (if needed) to your VMware image before it is converted and uploaded to AWS.

  1. Remove all snapshots – remove any snapshots that you may have. If you do not want to remove snapshots, create a copy of your .vmdk and create a new VMware image that points to it.
  2. Partition must be IDE – If you have any SCSI Devices, you should convert them to IDE. You may want to use a third party tool to do this such as Acronis.
  3. Uninstall VMware tools- if you have installed the VMWare tools, remove them.
  4. Enable Remote Desktop Access – you will want to enable remote desktop connections and make sure that they are permitted through the firewall (if enabled).

Convert VM to OVF

OVF (Open Virtual Format) is a universal virtual machine format that is used for distribution across organizations and products. You must convert your VMware image to this format so that AWS can convert it to its virtualization format. This process may take several hours depending on the size of the image.

    1. Download the too from: http://www.vmware.com/resources/techresources/1013
    2. Open a command line with administrator privileges
    3. Example Syntax:
      ovftool “C:\Virtual Machines\Windows Server 2008 x64.vmx” “C:\Virtual Machines\Windows Server 2008 x64.ovf”

Import to AWS, Setup & Connect

Once the VM has been converted to OVF format, it is ready to be imported to AWS.

  1. Open awsTools.bat
  2. Get Access Key and Secret Access Key from AWS: My Account / Console > Security Credentials
  3. Create Import instance using ec2-import-instance
    1. Example Syntax:
      ec2—import—instance “C:\Documents\Virtual Machines\Windows Server 2008 x64—disk1.vmdk” -f vmdk -t ml.large —a x86_64 —o XXXXXXXXXXXXXXXXXXXX —w XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX —b my_bucket
    2. b. -o = Access Key
    3. c. -w = Secret Access Key
    4. d. -t = Instance Types (see http://aws.amazon.com/ec2/instance-types/)
  4. Wait until the Instance is uploaded and converted
    1. Use ec2-describe-conversion-tasks to check status
  5.  Start Instance
  6. Open up RDP port (3389) using the AWS Security Group inbound rules
  7. Obtain an Elastic IP from AWS
  8. Under Network Interfaces in AWS, assign Elastic IP to the network interface that belongs to the instance
  9. Connect via RDP using Elastic IP address

References

Evens, S. (May 19, 2009). Setting up EC2 command line tools on Windows. Retrieved March 20, 2012 from http://sevans.info/2009/05/19/setting-up-ec2-command-line-tools-on-windows/.

matiasimc. (June 21, 2011). Re: VMWare Workstation VM import Windows 2008 R2 failing. Retrieved March 29, 2012, from https://forums.aws.amazon.com/thread.jspa?threadID=58171.

VMware. (May 21, 20009). VMware OVF Tool. Retrieved March 29, 2012, from http://www.vmware.com/resources/techresources/1013.