Installing OMS on CentOS 5.11

If your migrating old workloads to Azure occasionally you may not be able to use the latest Operating Systems. In these circumstances it may be useful to still be able to use the OMS agent to monitor your VM.

For CentOS – you really need to get to version 5.8 or above which has native support for Hyper-V, the Hypervisor that runs under the covers on Azure

Fix the CentOS repos to get the last released patches:

# cd /etc/yum.repos.d

edit CentOS-Base.repo and comment mirror list – change baseurl to start:

baseurl=http://vault.centos.org/5.11/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
baseurl=http://vault.centos.org/5.11/updates/$basearch/

set the date correctly & hostname ip in /etc/hosts

Get the repo for syslog (needed later)

# cd /etc/yum.repos.d
# wget http://rpms.adiscon.com/v8-stable/rsyslog.repo

Update the OS and install required software

# yum update
# yum groupinstall “Development Tools”
# yum groupinstall “Development Libraries”
# yum install python-ctypes
# yum install rsyslog

Update openssl (for TLS 1.2 support)

# wget https://www.openssl.org/source/openssl-1.0.2o.tar.gz —no-check-certificate
# tar -zxvf openssl-*.tar.gz
# cd openssl-*
# ./config -fpic shared && make && make install
# echo “/usr/local/ssl/lib” >> /etc/ld.so.conf$ ldconfig

Install updated version of wget

# wget http://ftp.gnu.org/gnu/wget/wget-1.19.5.tar.gz –no-check-certificate
# tar -xzvf wget-1.19.5.tar.gz
# cd wget-1.19.5
# make clean
# ./configure –with-ssl=openssl –with-libssl-prefix=/usr/local/ssl
# make && make install
# yum -y remove wget
# mv /usr/bin/wget /usr/bin/wget.orig
# ln -s /usr/local/bin/wget /usr/bin/wget

Install Updated version of Python

need late 2.7 (2.7.15) to support tls1.2

# wget https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz –no-check-certificate
# export CCFLAGS=”-I/usr/local/ssl/include/openssl”
# export LD_LIBRARY_PATH=”/usr/local/ssl/lib/”
# export LDFLAGS=”-L/usr/local/ssl/lib”
# make clean
# ./configure –with-ensurepip=install –prefix=/usr/local

The final result of ./configure –prefix=$DEPLOY may looks like follows:

Python build finished, but the necessary bits to build these modules were not found:
_tkinter bsddb185 dl
gdbm imageop sunaudiodev

To find the necessary bits, look in setup.py in detect_modules() for the module’s name.
It will show you the modules that can not be build, note that some of them are unnecessary or deprecated:

_tkinter: For tkinter graphy library, unnecessary if you don’t develop tkinter programs.
bsddb185: Older version of Oracle Berkeley DB. Undocumented. Install version 4.8 instead.
dl: For 32-bit machines. Deprecated. Use ctypes instead.
imageop: For 32-bit machines. Deprecated. Use PIL instead.
sunaudiodev: For Sun hardware. Deprecated

# make
# make altinstall

Check we have TLS 1.2 support

# python2.7
>>> import ssl
>>> print ssl.OPENSSL_VERSION
1.0.2

# pip2.7 uninstall cryptography
# pip2.7 install cryptography

Setup the python virtual environment

#/usr/local/bin/easy_install-2.7 virtualenv
# mkdir -p va/oms
# virtualenv va/oms
# cd va/oms/bin
# source activate

edit /etc/sysconfig/selinux
disabled

# wget https://github.com/Microsoft/OMS-Agent-for-Linux/releases/download/OMSAgent_v1.6.0-42/omsagent-1.6.0-42.universal.x64.sh –no-check-certificate

# ./omsagent-1.6.0-42.universal.x64.sh –install

(oms) [root@localhost bin]# ./omsadmin.sh -w 2 -s
………….
Starting Operations Management Suite agent (
info Configured omsconfig

Start OMS and check the logs for any problems

# /opt/microsoft/omsagent/bin/service_control restart
# tail -100 /var/opt/microsoft/omsagent/log/omsagent.log

clearing selinux messages:

# audit2allow -a
# audit2allow -a -M myallow
# semodule -i myallow.pp

Install updated curl Package

# mkdir ~/src/$ cd ~/src/
This installs curl in /usr/local/bin/curl
# cd /root/src$ wget http://curl.haxx.se/download/curl-7.42.1.tar.gz
# tar -xzvf curl-*.tar.gz
# cd curl-*
# ./configure –with-ssl=/usr/local/ssl –disable-ldap && make && make install

Install updated openssh

# wget https://mirror.bytemark.co.uk/pub/OpenBSD/OpenSSH/portable/openssh-7.7p1.tar.gz –no-check-certificate
# ./configure –with-ssl-dir=/usr/local/ssl
# make
# make install

Install Azure waagent

# wget https://github.com/Azure/WALinuxAgent/archive/master.zip
(oms) python setup.py install –register-service
# more /usr/lib/systemd/system/waagent.service
# python -u /usr/sbin/waagent -daemon

Azure to Terraform (az2tf)

‘Reverse Engineering’ Azure to Terraform

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. Terraform has great support for Azure, and it’s capabilities are being added to frequently see link

Configuration files describe to Terraform the components needed to run a single application or your entire Azure subscription. Terraform generates an execution plan describing what it will do to reach the desired state (terraform plan) , and then executes (terraform apply) it to build the described infrastructure in Azure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.

The above is taken form the Terraform website and Terraform is a great tool for creating new Azure Infrastructure and has some real advantages, notably the terraform plan capabilities and the simple elegant configuration language used to describe the infrastructure.

One problem with Terraform though is: What if you already have a Azure deployment you have built with ARM templates , PowerShell, cli2 or by just using the portal, how can you bring these under the control of terraform ?

Terraform Import Challenges

A partial answer to this problem is to use a “terraform import” command that allows you to import the current state of an Azure resource, but there are a few problems with this approach:

Multiple terraform imports have to be performed, referencing the correct full Azure resource ID.
You have to create a “stub” terraform configuration file for each of the resources before the import will work
The terraform stub file will have lots of information/parameters missing and does not represent your current infrastructure
You have to pay close attention to naming conventions as many Azure resources are interlinked. For example for a single VM you may have NIC(s), Public IP(s), NSG(s) , VNET, Subnet(s) , Route Table(s), Managed Disk(s) and A Storage Account for diagnostics. All these have to be consistently named as they will be cross-referenced in the terraform configuration files
A terraform plan command will likely show you there is much work to be done editing in all the missing information into your terraform configuration files.

The ‘az2tf’ Tool

What is needed is a tool that will read your current Azure deployment and automatically creates fully populated terraform configuration files and perform the corresponding ‘terraform import’ commands.

A subsequent ‘terraform plan’ command should report zero additions/deletions are required to the infrastructure and ideally zero ‘changes’ (The later is hard to achieve due to transient bugs and inconsistencies in how some string values are handled – but they are being worked through.).

This neutral output from the terraform plan command will give confidence that the automatically generated terraform configuration file are an accurate representation of the existing infrastructure within Azure.

A tool that can achieve the above called “az2tf” and is freely available on github here

This is written in a number “bash” shell scripts (one for each terraform provider) called from a master script ‘az2tf.sh’ and is intended to be run form the Terraform marketplace image:

https://docs.microsoft.com/en-us/azure/terraform/terraform-vm-msi

The tool is “use at your own risk” – but as it makes no changes to Azure and simply reads information , generates config files and runs a terraform plan it’s safe to use on any Azure subscription.

Using the ‘az2tf’ tool

Ensure you have setup and authorised both terraform and Azure cli2 correctly for your subscription (you’ll need read access – and also access (list, read) to any KeyVaults you use)

Download ‘az2tf’ or clone from github into an empty directory on your terraform vm

Run from the command line:

./az2tf.sh -s <your subscription id>

./az2tf.sh -s <your subscription id> -g <a resource group name>

Wait patiently – it’s slow – but good 🙂

As it runs it will:

Create a new sub directory tf.<your subscription id>
You’ll then see it looping around the terraform providers generating terraform config files and performing terraform imports.
For the final step it runs a terraform plan command

More details on the project page README.md on github

As this is a ‘hobby project’ there will be issues as a tool like this will need extensive testing to find all the edge cases, so check back regularly on github for updates and also for support of new azurerm terraform providers.

Part 2 (tbd) will go on the talk about how the tool was put together, using the primary tools – Azure cli2 ,bash (particularly jq and printf).

Calling Linux Custom Script Extensions from PowerShell

If you run Linux VM’s on Azure, then you at some point will want to call a Custom Script Extension that runs your own script (bash etc) to perform some operations within the virtual machine.

There are a few options for doing this including of course using chef and puppet, or as is documented here simply calling a Custom Script Extension with your own script.

When experimenting with doing this I found a lot of the documentation describing how to do this was out of date principally because it did not use the “customScript” extension type (publisher Microsoft.Azure.Extensions).

Note for Windows VM’s you use a different custom extension type “CustomScriptExtension” (publisher Microsoft.Compute) – see the commented section in the middle of the PowerShell script below

Listed below is the PowerShell to execute a command “command2.sh” which is stored as a blob in an Azure storage account in a container named “myscripts”. The PowerShell also grabs any output (stdout) from command2.sh – (getting stderr would be done in a similar way)

This script assumes you are using ARM and have previously logged into Azure from PowerShell (login-AzureRmAccount) and if required set the default subscription (Select-AzureRmSubscription)

$rg='your-resource-group'
$vmname='yourvm'
$storageaccountname='your-storage-account-name'
$cont='myscripts'
$Extensionname='customScript'
$vm = Get-AzurermVM -Name $vmname -ResourceGroupName $rg
# get the storage key
$key = (Get-AzureRmStorageAccountKey -Name $storageaccountname -ResourceGroupName $rg).value[0]
if (!$key) {
    write-output "Could not find a storage key"
    exit
}
#
# check if there's an existing custom script extension
# if there is remove it - your only allowed one at a time
#
$extname = ($VM.Extensions | Where { $_.VirtualMachineExtensionType -eq 'customScript' }).name
if ($extname) {
    write-output "removing existing extension: $extname"
    remove-azurermvmextension -name $extname  -ResourceGroupName $rg  -VMName $vmname -force
    write-output "removed - waiting 10 seconds ...."
    start-sleep -Seconds 10
}
# get extension types
# for windows use:
# Get-AzureRmVMExtensionImage -Location westeurope -PublisherName Microsoft.Compute -Type CustomScriptExtension
# for Linux use:
# Get-AzureRmVMExtensionImage -Location westeurope -PublisherName Microsoft.Azure.Extensions -Type customScript
#
#
#For Linux:
#
# Setup for call to Set-AzureRmExtension
#
$TheURI = "https://$storageaccountname.blob.core.windows.net/myscripts/command2.sh"
$Settings = @{"fileUris" = @($TheURI); "commandToExecute" = "./command2.sh"};
$ProtectedSettings = @{"storageAccountName" = $storageaccountname; "storageAccountKey" = $key};
#
Set-AzureRmVMExtension -ResourceGroupName $rg -Location $vm.location -VMName $vmname -Name "customScript" -Publisher "Microsoft.Azure.Extensions" -Type "customScript" -TypeHandlerVersion "2.0" -Settings $Settings -ProtectedSettings $ProtectedSettings
#
if ($?) {
  write-output "set extension ok"
  #
  # Get script extension output
  #
  $extout=((Get-AzureRmVM -Name $VMName -ResourceGroupName $RG -Status).Extensions | Where-Object {$_.Name -eq $ExtensionName}).statuses.Message
  #
  # Parse the stdout 
  #
  $stdout=$extout.substring($extout.indexof('[stdout]')+8,$extout.indexof('[stderr]')-$extout.indexof('[stdout]')-8)
  $stdout=$stdout.trim()
  write-output "stdout from command: $settings.commandToExecute"
  $stdout
  }
  else
  {
    write-output "set extension problem?"
  }
#
#

The above will be particularly useful when developing runbooks that are called as steps in a Azure Site Recovery plan.

If Multiple Linux VM’s are involved in the recovery plan it’s often necessary to query hostnames, assigned IP addresses or other information from Virtual Machine A and feed that information as parameter into Virtual Machine B etc. so it can correctly configure itself with Virtual Machine A’s information as part of the failover process.

Eg. As part of the failover plan – custom script extension 1 runs on VM1 (database server) and returns the hostname/ip address etc – which are then passed as parameters into custom extension script 2 run by VM2 (web server) which using the script reconfigures itself to use the new hostname/ip address given to the database server as it failed over into Azure, as this may be different to what was used ‘on-premise’ (or from the source Azure region if your using Azure to Azure ASR).

Part 4 – Testing the Network Triangulation

Having Completed parts 1,2 and 3 you should now have a functioning pair of links to both your chosen regions (West Europe and North Europe in my case.

In the Azure portal you should see 3 connections active on your VPN gateways – two are for the connections from West Europe to North Europe connectivity and one for the IPsec connection back to your VyOS:

4-0-az-gw-cnx

On your VyOS VM there are a few commands we can use to confirm all is as it should be

4-1-login-vyos

then check the IPsec tunnels are up:

$ show vpn ipsec sa

4-2-ipsec-up

You should see something like the above with your two public ip’s listed for the AZure VPN gateways in your two regions, both should be state “up” and some traffic flowing in/out

If you don’t see the above then you need yo go back and re-check your configuration so far and be sure the steps followed in parts 1-3 have been done correctly for your environment.

Sometimes it’s handy to reset the VPN on the VyOS , if your convinced that everything else is correct give this a try – edit a file (with vi) called for example cyclevpn.sh that has these commands within:

#/bin/vbash
source /opt/vyatta/etc/functions/script-template
run=/opt/vyatta/bin/vyatta-op-cmd-wrapper
$run restart vpn

then just run the command file you created:

$ ./cyclevpn.sh

Hopefully all else being correct your $ show vpn ipsec sa command will now show two functioning links as above.

If you have gotten this far it’s now time to see if BGP is functioning, run the command:

$ show ip bgp

4-3-bgp-sum

IN the out put above we are interested in a few things – firstly that we are getting route information from our two Azure BGP enabled gateways via BGP:

In the above we can see our two BGP (Azure) Neighbor’s 10.10.0.14 and 10.11.0.14, these IP’s were output by the PowerShell commands we ran in part 1 & 2 ($gateway1.BgpSettingsText and $gateway2.BgpSettingsText)

Notice how for both gateways we have received route prefix information shown by the “7” – if you don’t see numbers for both BGP neighbour IP’s then something is wrong in the configuration of either BGP or the IPsec tunnels may not be working correctly (see above).

Further down in the output we see the routing information and the chosen routes, looking at on example for subnet 10.10.1.0 – which is in the “spoke1” VNet in West Europe:

The two possible routes via 10.10.0.14 (West Europe) and 10.11.0.14 (North Europe) are shown. The route via 10.10.0.14 (West Europe) is preferred at this time as its ASPath (65010) via West Europe is shorter than the alternative path (65011, 65010) via North Europe then West Europe.

If you have gotten this far you should find your local VM can connect and ping to the VM in West Europe spoke successfully.

As a test you can now take down one of the two IPsec tunnels by deleting it:

$ conf
# del vpn ipsec site-to-site peer 1.2.3.4
# commit

(replace 1.2.3.4 with the public ip of the gateway in the region you want to disconnect from

The ping should continue uninterrupted, The output from the

“$ show ip bgp” command should now show a routing information with some route using a longer path to maintain the connectivity to your region, when I delete the IPsec tunnel to West Europe I see routing information like the following in the output from the command:

   Network         Next Hop           Metric LocPrf Weight Path
*> 10.10.0.0/24     10.10.0.14                             0 65010 65011 i
*> 10.10.1.0/24     10.10.0.14                             0 65010 65011 i
*> 10.10.2.0/24     10.10.0.14                             0 65010 65011 i
*> 10.11.0.0/24     10.10.0.14                             0 65011 I

This shows that my route to the West Europe is now longer – but connectivity is still maintained from my local CentOS VM to the VM in the spoke1 VNET in West Europe even though it’s IPsec tunnel is down.

Part 2 – Azure BGP Network Triangulation

In this post we’ll setup the Azure configuration in Region 2 (North Europe) and also setup the BGP Region 1 (West Europe) to Region 2 (North Europe) link.

As in part 1 you’ll need to create in advance, two VNets (vnspoke1 and vnhub0).The VNet vnspoke1 uses address space 10.11.1.0/24. Create a subnet within this VNet and place a VM within that we will use for our ping tests. Your setup should resemble to following:

The peering is setup in the same way as in part 1.

Here’s the PowerShell to create the North Europe Gateway and ready it for connection to our Home VyOS router.

# set some variables for Resource Group, VNet Name, Location, Name Azure

# Gateway, Name of our our local home network definition, The private ASN number to use # for the Azure West Europe network, The private ASN number to use for our Home

# network, The BGP address used at home, the subnet range for BGP

$rg=”atnettest2″
$vn=”vnhub0″
$loc=’northeurope’
$vngwname=’gwnebgp’
$gwlocal=’homenet’
$VNet1ASN=65011
# Private ASN for home network
$LNGASN=65168
# these BGP addresses come from part 3
$BGPPeerIP5=’10.168.0.253′
$locprefix=’10.168.0.253/32′

# get the vnet and gateway subnet
$vnet = Get-AzureRmVirtualNetwork -ResourceGroupName $rg -Name $vn
$gwpip= New-AzureRmPublicIpAddress -Name gwpip -ResourceGroupName $rg -Location $loc -AllocationMethod Dynamic
$subnet = Get-AzureRmVirtualNetworkSubnetConfig -Name ‘GatewaySubnet’ -VirtualNetwork $vnet

# get a public ip for the gateway
$gwipconfig = New-AzureRmVirtualNetworkGatewayIpConfig -Name gwipconfig1 -SubnetId $subnet.Id -PublicIpAddressId $gwpip.Id
#
# make the gateway – will take a while – typically about 30 minutes
#
date
#New-AzureRmVirtualNetworkGateway -Name $vngwname -ResourceGroupName $rg -Location $loc -IpConfigurations $gwipconfig -GatewayType Vpn -VpnType RouteBased -GatewaySku Standard -Asn $VNet1ASN -EnableBgp $true
date
# get the public ip for local gw we’ll need to make a note of this for part 3
Get-AzureRmPublicIpAddress -Name gwpip -ResourceGroupName $rg
$gateway1 = Get-AzureRmVirtualNetworkGateway -Name $vngwname -ResourceGroupName $rg

# get the BGP ip for local gw – we’ll need to make a note of this for part 3
$gateway1.BgpSettingsText

# In the following replace a.b.c.d in the following with your home public IP address
New-AzureRmLocalNetworkGateway -Name $gwlocal -ResourceGroupName $rg -Location $loc -GatewayIpAddress ‘81.108.251.116’ -AddressPrefix $locprefix -Asn $LNGASN -BgpPeeringAddress $BGPPeerIP5
## get local gateway and on-prem local info
$local = Get-AzureRmLocalNetworkGateway -Name $gwlocal -ResourceGroupName $rg
## connect the two
New-AzureRmVirtualNetworkGatewayConnection -Name northeurtohome -ResourceGroupName $rg -Location $loc -VirtualNetworkGateway1 $gateway1 -LocalNetworkGateway2 $local -ConnectionType IPsec -RoutingWeight 10 -SharedKey ‘abc1234’-enablebgp $true

Now setup the gateway to gateway BGP link:

inter-bgp

# set some variables

$rg1=”atnettest”
$rg2=”atnettest2″
$loc1=’westeurope’
$loc2=’northeurope’
$vngwname1=’gwwebgp’
$vngwname2=’gwnebgp’
$Connection12 = “wetone”
$Connection21 = “netowe”
#get the West Europe Gateway
$gateway1 = Get-AzureRmVirtualNetworkGateway -Name $vngwname1 -ResourceGroupName $rg1

# check BGP on on
$gateway1.BgpSettingsText

#get the North Europe Gateway
$gateway2 = Get-AzureRmVirtualNetworkGateway -Name $vngwname2 -ResourceGroupName $rg2

# check BGP ip address
$gateway2.BgpSettingsText

# Create the links (two are needed)
New-AzureRmVirtualNetworkGatewayConnection -Name $Connection12 -ResourceGroupName $rg1 -VirtualNetworkGateway1 $gateway1 -VirtualNetworkGateway2 $gateway2 -Location $loc1 -ConnectionType Vnet2Vnet -SharedKey ‘AzureA1b2C3’ -EnableBgp $True -RoutingWeight 10
New-AzureRmVirtualNetworkGatewayConnection -Name $Connection21 -ResourceGroupName $rg2 -VirtualNetworkGateway1 $gateway2 -VirtualNetworkGateway2 $gateway1 -Location $loc2 -ConnectionType Vnet2Vnet -SharedKey ‘AzureA1b2C3’ -EnableBgp $True -RoutingWeight 10

Part 1 – Azure BGP Network Triangulation

In this post we’ll setup the Azure configuration in Region 1 (West Europe)

You will need to create in advance, two VNets (vnspoke1 and vnhub0).

The VNet vnspoke1 uses address space 10.10.1.0/24. Create a subnet within this VNet and place a VM within that we will use for our ping tests. Your setup should resemble to following:

As VNet peering is a relatively new capability within Azure, for clarity are the network peering setting from the portal for vnspoke1:

and vnhub0:

The VNet vnhub0 uses address space 10.10.0.0/24 within this create a default gateway subnet that will hold our Virtual Network Gateway “gwwebgp”. This PowerShell sets up this gateway

# set some variables for Resource Group, VNet Name, Location, Name Azure

# Gateway, Name of our local home network definition, The private ASN number to use

# for the Azure West Europe network, The private ASN number to use for our Home

# network, The BGP address used at home, the subnet range for BGP
$rg=”atnettest”
$vn=”vnhub0″
$loc=’westeurope’
$vngwname=’gwwebgp’
$gwlocal=’homenet’
$VNet1ASN=65010

# Private ASN for home network
$LNGASN=65168

# these BGP addresses come from part 3
$BGPPeerIP5=’10.168.0.253′
$locprefix=’10.168.0.253/32′

# assumes VNet vnhub0 & gateway subnet already created
$vnet = Get-AzureRmVirtualNetwork -ResourceGroupName $rg -Name $vn
$gwpip= New-AzureRmPublicIpAddress -Name gwpip -ResourceGroupName $rg -Location $loc -AllocationMethod Dynamic
$subnet = Get-AzureRmVirtualNetworkSubnetConfig -Name ‘GatewaySubnet’ -VirtualNetwork $vnet

# get a public ip for the gateway
$gwipconfig = New-AzureRmVirtualNetworkGatewayIpConfig -Name gwipconfig1 -SubnetId $subnet.Id -PublicIpAddressId $gwpip.Id
#
# make the gateway – will take a while – typically about 30 minutes
#
date
New-AzureRmVirtualNetworkGateway -Name $vngwname -ResourceGroupName $rg -Location $loc -IpConfigurations $gwipconfig -GatewayType Vpn -VpnType RouteBased -GatewaySku Standard -Asn $VNet1ASN -EnableBgp $true
#New-AzureRmVirtualNetworkGateway -Name $vngwname -ResourceGroupName $rg -Location $loc -IpConfigurations $gwipconfig -GatewayType Vpn -VpnType RouteBased -GatewaySku HighPerformance -Asn $VNet1ASN -EnableBgp $true
date

# get the public ip for local gw we’ll need to make a note of this for part 3
Get-AzureRmPublicIpAddress -Name gwpip -ResourceGroupName $rg
$gateway1 = Get-AzureRmVirtualNetworkGateway -Name $vngwname -ResourceGroupName $rg

# get the BGP ip for local gw – we’ll need to make a note of this for part 3
$gateway1.BgpSettingsText
#
# In the following replace a.b.c.d in the following with your home public IP address
New-AzureRmLocalNetworkGateway -Name $gwlocal -ResourceGroupName $rg

-Location $loc -GatewayIpAddress ‘a.b.c.d’ -AddressPrefix $locprefix -Asn $LNGASN -BgpPeeringAddress $BGPPeerIP5
# get local network gateway
$local = Get-AzureRmLocalNetworkGateway -Name $gwlocal -ResourceGroupName $rg
# connect the two
New-AzureRmVirtualNetworkGatewayConnection -Name westeurtohome -ResourceGroupName $rg -Location $loc -VirtualNetworkGateway1 $gateway1 -LocalNetworkGateway2 $local -ConnectionType IPsec -RoutingWeight 10 -SharedKey ‘abc1234’-enablebgp $true

This should leave you with a Standard BGP enabled gateway in VNet vnhub0:

Part 3 – Azure BGP Network Triangulation

In this part we’ll setup the VyOS VM and configure the required networking options for IPsec and BGP within VyOS.

home

Home Network:

Like many folks my home network has a private IP address space (192.168.0.0/24) and my broadband provider supplies a single public IP address a.b.c.d. If you’d like to know you public IP address look on your router or go to a browser on your home machine and navigate to http://whatsmyip.org or http://whatsmyip.net

My home router NAT’s internal private address space onto the public IP.

I have a small server the private network running HyperV within this I create two VM’s

One for VyOs and one for VM2 running a base install of CentOS

Before starting to configure the VyOS VM download the VyOS .iso from the download section of http://vyos.net. At the time of writing I used this 64bit stable release:
http://packages.vyos.net/iso/release/1.1.7/vyos-1.1.7-amd64.iso

The VyOS VM needs:
1GB RAM, 1cpu, 2GB disk drive and 3 Network Adapters: 1 external and 2 internal

hyperv-home

The VM should be set to boot off the VyOS iso image loaded to the virtual CD

Boot the VM off the iso, when presented with the login prompt enter “vyos” for the username and password, then:

$ install image
… installation stuff …
$ reboot

After the reboot we need to configure the VyOS for both IPsec and BGP setup so it will communicate with our two gateways we setup in part 1 and part 2

This first part is standard stuff much of this may already be setup login to the VyOS VM and enter configure mode

vyos-login

and the enter the commands below

set service ssh port ’22’
set system config-management commit-revisions ’20’
set system console device ttyS0 speed ‘9600’
set system host-name ‘vyos’
set system login user vyos level ‘admin’
set system ntp server ‘0.pool.ntp.org’
set system ntp server ‘1.pool.ntp.org’
set system ntp server ‘2.pool.ntp.org’
set system package auto-sync ‘1’
set system package repository community components ‘main’
set system package repository community distribution ‘helium’
set system package repository community password ”
set system package repository community url ‘http://packages.vyos.net/vyos’
set system package repository community username ”
set system syslog global facility all level ‘notice’
set system syslog global facility protocols level ‘debug’
set system time-zone ‘UTC’

Next we define the 3 interfaces we are going to use

eth0 – connected to our external network (the internet), this is given a static IP address by my home router which always maps the MAC address ’00:15:5d:00:2b:06′ to the IP 192.168.0.197. This same IP address is used in the DMZ in the router

eth1 – Our internal network – set to 10.168.0.1

eth1 -address for BGP – set to 10.168.0.253/32 (a single IP net)

set interfaces ethernet eth0 address ‘dhcp’
set interfaces ethernet eth0 description ‘OUTSIDE’
set interfaces ethernet eth0 duplex ‘auto’
set interfaces ethernet eth0 hw-id ’00:15:5d:00:2b:06′
set interfaces ethernet eth0 smp_affinity ‘auto’
set interfaces ethernet eth0 speed ‘auto’
set interfaces ethernet eth1 address ‘10.168.0.1/24’
set interfaces ethernet eth1 description ‘INSIDE’
set interfaces ethernet eth1 duplex ‘auto’
set interfaces ethernet eth1 hw-id ’00:15:5d:00:2b:08′
set interfaces ethernet eth1 smp_affinity ‘auto’
set interfaces ethernet eth1 speed ‘auto’
set interfaces ethernet eth2 address ‘10.168.0.253/32’
set interfaces ethernet eth2 duplex ‘auto’
set interfaces ethernet eth2 hw-id ’00:15:5d:00:2b:0a’
set interfaces ethernet eth2 smp_affinity ‘auto’
set interfaces ethernet eth2 speed ‘auto’
set interfaces loopback ‘lo’

Set up the IPsec preamble for link West Europe’s gateway

set vpn ipsec esp-group atazure compression ‘disable’
set vpn ipsec esp-group atazure lifetime ‘3600’
set vpn ipsec esp-group atazure mode ‘tunnel’
set vpn ipsec esp-group atazure pfs ‘disable’
set vpn ipsec esp-group atazure proposal 1 encryption ‘aes256’
set vpn ipsec esp-group atazure proposal 1 hash ‘sha1’
set vpn ipsec ike-group atazure-ike ikev2-reauth ‘no’
set vpn ipsec ike-group atazure-ike key-exchange ‘ikev2’
set vpn ipsec ike-group atazure-ike lifetime ‘10800’
set vpn ipsec ike-group atazure-ike proposal 1 dh-group ‘2’
set vpn ipsec ike-group atazure-ike proposal 1 encryption ‘aes256’
set vpn ipsec ike-group atazure-ike proposal 1 hash ‘sha1’

Set up the IPsec preamble for link North Europe’s gateway

set vpn ipsec esp-group atazurene compression ‘disable’
set vpn ipsec esp-group atazurene lifetime ‘3600’
set vpn ipsec esp-group atazurene mode ‘tunnel’
set vpn ipsec esp-group atazurene pfs ‘disable’
set vpn ipsec esp-group atazurene proposal 1 encryption ‘aes256’
set vpn ipsec esp-group atazurene proposal 1 hash ‘sha1’
set vpn ipsec ike-group atazurene-ike ikev2-reauth ‘no’
set vpn ipsec ike-group atazurene-ike key-exchange ‘ikev2’
set vpn ipsec ike-group atazurene-ike lifetime ‘10800’
set vpn ipsec ike-group atazurene-ike proposal 1 dh-group ‘2’
set vpn ipsec ike-group atazurene-ike proposal 1 encryption ‘aes256’
set vpn ipsec ike-group atazurene-ike proposal 1 hash ‘sha1’

set vpn ipsec ipsec-interfaces interface ‘eth0’
set vpn ipsec nat-traversal ‘enable’

Initiate tunnel to North Europe:

set vpn ipsec site-to-site peer p.q.r.s authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer p.q.r.s authentication pre-shared-secret ‘abc1234’
set vpn ipsec site-to-site peer p.q.r.s connection-type ‘initiate’
set vpn ipsec site-to-site peer p.q.r.s default-esp-group ‘atazurene’
set vpn ipsec site-to-site peer p.q.r.s description ‘VNet GW Az Nrth Eur’
set vpn ipsec site-to-site peer p.q.r.s ike-group ‘atazurene-ike’
set vpn ipsec site-to-site peer p.q.r.s ikev2-reauth ‘inherit’
set vpn ipsec site-to-site peer p.q.r.s local-address ‘192.168.0.197’
set vpn ipsec site-to-site peer p.q.r.s tunnel 1 allow-nat-networks ‘disable’
set vpn ipsec site-to-site peer p.q.r.s tunnel 1 allow-public-networks ‘disable’
set vpn ipsec site-to-site peer p.q.r.s tunnel 1 local prefix ‘10.168.0.0/24’
set vpn ipsec site-to-site peer p.q.r.s tunnel 1 remote prefix ‘10.0.0.0/8’

Initiate tunnel to West Europe:

set vpn ipsec site-to-site peer w.x.y.z authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer w.x.y.z authentication pre-shared-secret ‘abc1234’
set vpn ipsec site-to-site peer w.x.y.z connection-type ‘initiate’
set vpn ipsec site-to-site peer w.x.y.z default-esp-group ‘atazurene’
set vpn ipsec site-to-site peer w.x.y.z description ‘VNet GW Az Nrth Eur’
set vpn ipsec site-to-site peer w.x.y.z ike-group ‘atazurene-ike’
set vpn ipsec site-to-site peer w.x.y.z ikev2-reauth ‘inherit’
set vpn ipsec site-to-site peer w.x.y.z local-address ‘192.168.0.197’
set vpn ipsec site-to-site peer w.x.y.z tunnel 1 allow-nat-networks ‘disable’
set vpn ipsec site-to-site peer w.x.y.z tunnel 1 allow-public-networks ‘disable’
set vpn ipsec site-to-site peer w.x.y.z tunnel 1 local prefix ‘10.168.0.0/24’
set vpn ipsec site-to-site peer w.x.y.z tunnel 1 remote prefix ‘10.0.0.0/8’

Default route – and blackhole route for BGP and set private ASN number

set protocols static route 0.0.0.0/0 next-hop ‘192.168.0.1’
set protocols static route 10.168.0.0/24 ‘blackhole’
set protocols bgp 65168 network ‘10.168.0.0/24’

BGP for West Europe

set protocols bgp 65168 neighbor 10.10.0.14 ebgp-multihop ‘8’
set protocols bgp 65168 neighbor 10.10.0.14 remote-as ‘65010’
set protocols bgp 65168 neighbor 10.10.0.14 soft-reconfiguration ‘inbound’

BGP for North Europe

set protocols bgp 65168 neighbor 10.11.0.14 ebgp-multihop ‘8’
set protocols bgp 65168 neighbor 10.11.0.14 remote-as ‘65011’
set protocols bgp 65168 neighbor 10.11.0.14 soft-reconfiguration ‘inbound’

Having don’t the above you should be set for some testing of the connectivity which is described in part 4

Azure BGP Network Triangulation (from your home!)

In this post I’ll show how to setup two Azure BGP gateways in different locations and a third BGP gateway at Home using VyOS. (a community fork of Vyatta)

VyOS is an open source network operating system that can be installed on physical hardware or a virtual machine on your own server (or laptop) at home (or on premise)

Using VyOS allows experimentation with some more advanced networking features such as IPsec and BGP routing that would normally require a device such as a Cisco router other hardware options.

VyOS can be run in a small Hyper-V (or VirtualBox) VM on your own laptop and provided you have DMZ capabilities on your home internet router (most do) from your home network.

In this lab we’ll setup something that looks like this:

route1

We’ll get VM1 pinging VM2

Then we’ll deliberately break one of the IPsec links and show that the connectivity remains and is automatically handled by the BGP routing adjustments to use an alternative route:

route2

We’ll break this down into four sections:

Azure VM Network Bandwidth

July 2017 – Updated with new VPN Gateway types

In this blog post we look at some network bandwidth tests for a variety of Azure VM sizes.

The tests have been run between two VM’s in the same VNet. Network bandwidth testing has been done with Linux using iperf3 running on CentOS 7.2 and Windows 2016 using the Ntttcp tool.

nettest1

Both single stream and multi stream tests were used. Of course your actual throughput numbers will vary from the ones seen in the tests below due to a number of factors (OS Type, workload characteristics etc….)

Note: Microsoft is currently in the process of implementing some Azure Network optimisations:

1. “Receive Side Scaling” – https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-optimize-network-bandwidth
2. “Accelerated Networking” – https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-accelerated-networking-portal

The table below has been updated to include optimization 1. further updates will follow later in the year.
Entries in green include optimisation 1 “Receive Side Scaling”.

OS	VM Type	# Cores	Single Stream Throughput	‘N’ Streams Throughput (1x #Cores)
CentOS (with 1)	A0 Basic	0.25	10 Mbps	10 Mbps
CentOS (with 1)	A1 Basic	1	100 Mbps	100 Mbps
CentOS (with 1)	A2 Basic	2	200 Mbps	200 Mbps
CentOS (with 1)	A3 Basic	4	400 Mbps	395 Mbps
CentOS (with 1)	A4 Basic	8	805 Mbps	816 Mbps
Windows	A4 Basic	8	737 Mbps	734 Mbps
CentOS (with 1)	A1 Standard	4	500 Mbps	500 Mbps
CentOS (with 1)	A2 Standard	4	500 Mbps	500 Mbps
CentOS (with 1)	A3 Standard	4	1000 Mbps	1000 Mbps
CentOS (with 1)	A4 Standard	8	1990 Mbps	2000 Mbps
CentOS (with 1)	A6 Standard	8	1000 Mbps	1000 Mbps
CentOS (with 1)	A6 Standard	8	2000 Mbps	2000 Mbps
CentOS (with 1)	A1 v2	1	495 Mbps	488 Mbps
Windows	A1 v2	1	411 Mbps	467 Mbps
CentOS (with 1)	A2 v2	2	500 Mbps	492 Mbps
CentOS (with 1)	A4 v2	4	998 Mbps	999 Mbps
CentOS (with 1)	A8 v2	8	1980 Mbps	1910 Mbps
CentOS (with 1)	A8	8	4000 Mbps	4200 Mbps
CentOS (with 1)	A9	16	4550 Mbps	7850 Mbps
CentOS (with 1)	A10	8	3990 Mbps	3995 Mbps
Windows	A10	8	1820 Mbps	3942 Mbps
CentOS (with 1)	A11	16	4410 Mbps	8000 Mbps
CentOS (with 1)	D1 v2	1	750 Mbps	726 Mbps
CentOS (with 1)	D2 v2	2	1500 Mbps	1500 Mbps
CentOS (with 1)	D3 v2	4	3000 Mbps	3000 Mbps
CentOS (with 1)	D4 v2	8	4950 Mbps	6000 Mbps
CentOS (with 1)	D5 v2	16	4840 Mbps	12000 Mbps
CentOS (with 1)	D11 v2	2	1500 Mbps	1500 Mbps
CentOS (with 1)	D12 v2	4	3000 Mbps	3000 Mbps
CentOS (with 1)	D13 v2	8	4210 Mbps	5990 Mbps
CentOS (with 1)	D14 v2	16	4990 Mbps	11900 Mbps
CentOS (with 1)	D15 v2	20	4440 Mbps	15500 Mbps
Windows	D15 v2	20	1002 Mbps	12176 Mbps
CentOS (with 1)	F1	1	750 Mbps	749 Mbps
CentOS (with 1)	F2	2	1500 Mbps	1490 Mbps
CentOS (with 1)	F4	4	2990 Mbps	2995 Mbps
Windows (with 1)	F4	4	928 Mbps	2640 Mbps
CentOS (with 1)	F8	8	3490 Mbps	5990 Mbps
Windows	F8	8	390 Mbps	4388 Mbps
CentOS (with 1)	F16	16	4110 Mbps	11800 Mbps
Windows (with 1)	F16	16	1096 Mbps	8416 Mbps
CentOS (with 1)	G1	2	2000 Mbps	2000 Mbps
CentOS (with 1)	G2	4	3270 Mbps	4000 Mbps
CentOS (with 1)	G3	8	3160 Mbps	8000 Mbps
CentOS (with 1)	G4	16	3970 Mbps	8880 Mbps
Windows	G4	16	1602 Mbps	9488 Mbps
Windows (with 1)	G4	16	1904 Mbps	7856 Mbps
CentOS (with 1)	G5	32	3850 Mbps	13700 Mbps
CentOS (with 1)	NV6	6	4850 Mbps	5970 Mbps
CentOS (with 1)	NV12	12	4760 Mbps	12200 Mbps

Test method

In each CentOS VM:

$ sudo yum -y update (a very important step !)

Ensure Receive Side Scaling is enabled see: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-optimize-network-bandwidth

$ wget https://iperf.fr/download/fedora/iperf3-3.1.3-1.fc24.x86_64.rpm
$ sudo yum install iperf3-3.1.3-1.fc24.x86_64.rpm

On one VM run iperf in server mode
$ iperf3 -s

On another VM run single stream test:
$ iperf3 -c ip-of-server

For the multiple streams test:
$ iperf3 -c ip-of-server -P n

Where n = number of cores in VM

In each Windows 2016 VM:

Apply the latest Windows updates then download ntttcp from here

On one VM (ip=w.x.y.z) run ntttcp in receiver mode
C:> ntttcp -r -m 1,*,w.x.y.z
&
C:> ntttcp -r -m n,*,w.x.y.z (for the multi-thread test)
Where n = 8x number of cores in VM

On another VM run single thread test ntttcp in sender mode:
C:> ntttcp -s -m 1,*,w.x.y.z.

For the multi thread test:
C:> ntttcp -s -m n,*,w.x.y.z.

Where n = the number of cores in VM

Peering VNets (Directly connected)

nettest2

Testing between VM’s in directly peered VNets showed no noticeable difference.

Peered VNets in the same region

Testing between VM’s indirectly peered via the new gateway types (VPMGW1, 2 & 3) shows bandwidth up to 2.8Gbs a big step forward from the previous gateway types that returned a maximum of 980 Mbps when using the now depreciated ‘High Performance’ gateway.

nettest3

Gateway Type	Single Stream Throughput	‘N’ Stream Throughput (1x #Cores)
VPNGW1	700 Mbps	717 Mbps
VPNGW2	1400 Mbps	1430 Mbps
VPNGW3	1810 Mbps	2880 Mbps

For comparison here are the results form the now depreciated gateway types Standard and High Performance:

Gateway Type	Single Stream Throughput	‘N’ Streams Throughput (1x #Cores)
Standard	472Mbps	580 Mbps
High Performance	720 Mbps	980 Mbps

Peered VNets via two BGP Gateways one in each region

nettest4

Gateway Type	Single Stream Throughput	‘N’ Streams Throughput
VPNGW1	550 Mbps	670 Mbps
VPNGW2	650 Mbps	780 Mbps
VPNGW3	650 Mbps	650 Mbps

For comparison here are the results form the now depreciated gateway types Standard and High Performance:

Gateway Type	Single Stream Throughput	‘N’ Streams Throughput
Standard	200 Mbps	280 Mbps
High Performance	210 Mbps	411 Mbps

Azure Application Proxy – Part 4

Setting up Azure Active Directory Sync

The first step is to add a “custom domain” to Azure Active Directory in preparation for directory synchronization.

In this sequence we will add our custom domain (in my case “andyt.work”) to our default directory we were assigned when we created a new Azure account.

Select Active Directory from the left hand icons and then ensure the Default Directory is selected.

appproxyp4-0-admin

Now ensure one of your cloud user has “Global Admin” rights over the directory, here I have a user called “admin” that has that capability:

appproxyp4-1-global

Now enter the “Domains” section of the Default Directory and Click the Add button at the bottom of the screen.

appproxyp4-2-add

Enter the domain name you own and click “add” appproxyp4-3-andyt

The next step involves making a TX record entry with domain registrar. The details of the TX record are shown on screen. There is also a link to a web page describing the process of adding the required TX record for various different domain registrars.

Once you have added the TX record and waited a while for it to propagate click “Verify”

appproxyp4-4-verify

Having completed the verify process you should see the domain as verified , you can now switch the assigned primary domain to your custom domain using the “Change Primary” button at the bottom of the screen.

appproxyp4-5-chpri

Specify your domain as the new primary and press the tick button to confirm

appproxyp4-6-chpri2

We now have registered our domain and made it the primary.

The next step is to synchronize our on premise active directory domain with the Azure hosted Active Directory.

Navigate to the Directory Integration section of the Default Directory and enable Directory Sync, and Save

appproxyp4-7-activate

Next download the AD connect tool to one of your domain controllers or a domain connected machine/VM from:

https://www.microsoft.com/en-us/download/details.aspx?id=47594

Run the Wizard

appproxyp4-8-exe

The AD Connect Wizard has an Express set up option which is sufficient for most cases

appproxyp4-9-wiz1

Enter the credentials of your Azure user that has Global Admin rights that we set up earlier (“admin” in my case).

appproxyp4-12-wiz4

Enter credentials of the local domain administrator

appproxyp4-13-wiz5

Finish by clicking Install

appproxyp4-14-wiz6

Back in the Azure portal you should now see any local users defined in your Active Directory domain also appearing in the users section of the Default Directory:

appproxyp4-16-synched

Your directory is now “synced” with Azure Active Directory, users can sign on to cloud based application using their username (email address) and the same password as they use on premise to sign onto your local domain.