A few weeks ago we had a requirement to restrict the outbounds ports of HDinsight for security reasons, so this article is dedicated to that requirement. Before we begin Microsoft official position on this is:

Important: HDInsight doesn’t support restricting outbound traffic, only inbound traffic. When defining Network Security Group rules for the subnet that contains HDInsight, only use inbound rules.

So after reading the above (from: https://azure.microsoft.com/en-gb/documentation/articles/hdinsight-extend-hadoop-virtual-network/) we took it as a challenge to get this working, after much testing we managed to get all the required ports. We have tried by deploying multiple clusters and so far it all works and deploys correctly, a couple of notes:

• The solution below is not 100% secure but it mitigates the risk by lowering the “attack” service to only the regional azure IPs.
• We also needed to open port 80 to the Ubuntu website (91.189.88.0/21) as this is required by some of the Apache tests after deployment
• While testing we noticed that the servers communicate with the management point over a random port, this port seemed to be in the same range as the dynamic Azure SQL ports of 11000-11999 and 14000-14999. However to be on the safe side we opened a larger range 10000-49151 as we can’t be 100% sure.
• You will need to open multiple rules for each Azure Regional IP (i suggest you combine the ips to the second octed). The ip addresses can be found here: https://www.microsoft.com/en-gb/download/details.aspx?id=41653. You will also need to keep the ip addresses updated (A new xml file will be uploaded every Wednesday (Pacific Time) with the new planned IP address ranges. New IP address ranges will be effective on the following Monday (Pacific Time)).
• This is all unofficial and while we have had no problems with multiple deployments i can’t give any guarantees.

Inbound Ports

Outbound Ports

Update 19/02/2017

Microsoft has now introduced a new button which updates all modules for you, very easy as shown on the screenshot:

-== Outdated ==-

Now if any of you use Azure Automation you know that updating the Powershell Modules is a pain as they require dependency to be installed first.This can easily take you a whole day to do by hand. However there is a very easy way to this.

The way to update them all including dependency is to run the json template for the AzureRM module. You can find the deploy to azure button and the lates version here: https://www.powershellgallery.com/packages/AzureRM/. All you need to do is chose the subscription, resource and your automation account.

Please note that the file currently does not include all regions and as such you may need to do it manually as described below:

You may need to modify it as the file does not include all regions where automation is available and will fail. You can use the code below to replace the json Automation Account Location section before you run the template:

Updated 08/01/2017

Due to the lack of articles regarding this topic i decided to do a quick post on how to get the Custom Script extension to work correctly on both Linux and Windows ARM (Resource Manager) virtual machines.

Some very important notes and key differences before we get started

• For Windows machines
  • The extension details are:
    • $ExtensionType = ‘CustomScriptExtension’
    • $Publisher = ‘Microsoft.Compute’
    • $Version = ‘1.8’
  • When entering the commandToExecute note that in Windows the command is executed from the root of the container. This means that if your script in located in “\scripts\version1\my.ps1” on the blob storage container to run the powershell script you need to reference the full path like shown below. This is because when the agent downloads the files it creates the folder structure the same way as the blob (Do not put the container name in the path!):
    
    1	"commandToExecute" = "powershell.exe -ExecutionPolicy Unrestricted -File '\scripts\version1\my.ps1'"

  • If you get the directory path wrong this will be indicated by an error in the logs located at “C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Status”. The error will be something like:
    
    1	The term '.\\scripts\\myscript.ps1' is not \\nrecognized as the name of a cmdlet, function, script file, or operable \\nprogram. Check the spelling of the name, or if a path was included, verify \\nthat the path is correct and try again.

  • The download directory is located in “C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Downloads”
  • Log file are in “C:\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtension\1.8”. Be aware they are a bit generic for more details errors including errors generated by your script inside the machine by not using throw please check: “C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Status”
  • If you are using named parameters rather than arguments in your script “eg. -MemoryinGB 100” you need to use the -Command parameter rather than -File, like:
    
    1	"commandToExecute" = "powershell.exe -ExecutionPolicy Unrestricted -Command '\scripts\version1\my.ps1 -MemoryinGB 100 - Name Archi'"

  • If you want your script to tell Azure automation that an error has occurred that’s meaningful make sure you use “Throw” in your script (the one that runs on the machine) as the exit, if you want to catch all messages regardless if they use throw make sure you use try,catch and finally as shown below, this will give you the error messages from “C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.8\Status” 
  • To get the status you can use the command below. Just specify the virtual machine name, it’s resource group and the name you gave to your custom extension:
    
    1	((Get-AzureRmVM -Name $VMName -ResourceGroupName $RGName -Status).Extensions | Where-Object {$_.Name -eq $ExtensionName}).Substatuses

• For all Machines
  • Any files downlaoded are not removed after script has finished executing
  • You can use the “Get-AzureServiceAvailableExtension” to get all the extensions and there current version
  • If you are using Azure Automation scripts and you would like them to fail correctly make sure you add the line below at the top of your code :
    
    1	$ErrorActionPreference = "Stop"

  • You can only run a script with the same parameters on a virtual machine once. If yo want to run it multiple times on the same VM you can specify a time stamp in the Setting or SettingString of Set-AzureRmVMExtension
    
    1 2	$timestamp = (Get-Date).Ticks "timestamp" = "$timestamp";

  • If you are using hashtables (and i recommend them) note that a certain format is expected for the fileUris in the Setting of Set-AzureRmVMExtension. Since we can get the extension to download multiple files for us we need to follow the following format:
    • For a Single File
      
      1	"fileUris" = [Object[]]"http://example.com/showmemore.sh";

    • For Multiple Files
      
      1	"fileUris" = [Object]"http://example.com/showmemore.sh","http://example.com/otherthing.sh";

  • If you want to execute commands with sensitive variables like passwords you can move the commandToExecute to ProtectedSettings or ProtectedSettingString in Set-AzureRmVMExtension. Make sure you only have it in one place (Setting or ProtectedSetting).
• For Linux Machines
  • The extension details are:
    • $ExtensionType = ‘CustomScriptForLinux’
    • $Publisher = ‘Microsoft.OSTCExtensions’
    • $Version = ‘1.5’
  • When entering the commandToExecute note that in Linux the command is executed from the same folder where the script is located. This is because all files are downloaded to the same folder and blob folder structure is ignored. This means that if your script in located in “\scripts\version1\” on the blob storage container to run the sh script you need to ignore the structure and only specify the file name like:
    
    1	"commandToExecute" = "sh showmemore.sh"

  • The download directory is located in “/var/lib/waagent/Microsoft.OSTCExtensions.CustomScriptForLinux-1.5.2.0/download/”
  • Log file are in “/var/log/azure/Microsoft.OSTCExtensions.CustomScriptForLinux/1.5.2.0”
  • If you have your own DNS servers and you haven’t set it up to forward for Azure dns queries you might get an error. If you run “hostname -f” and you get errors, you can tell the custom script to skip dns check with the code below in the Setting or SettingString of Set-AzureRmVMExtension. Note that at this stage the script wants a bool variable, looking at the code future versions will take a string.
    
    1	"enableInternalDNSCheck" = $False

  • To get the status you can use the command below. Just specify the virtual machine name, it’s resource group and the name you gave to your custom extension:
    
    1	((Get-AzureRmVM -Name $VMName -ResourceGroupName $RGName -Status).Extensions | Where-Object {$_.Name -eq $ExtensionName}).Statuses

• Two important Notes:
  • You can only have one custom extension at a time so you can remove it after you run your script or before with a code similar to the one below:
  • Currently, if the extension is installed it will be re-run every time the VM is Delocated and Started again, to avoid this remove the extension  with a code similar to the one below:

So let’s see some scripts the first is a Windows script which create a dns entry on a Domain Controller:

And here is a Linux script which join a Linux machine to a domain, however this time we don’t want the execution of the script to log the variables we pass to it so we move it to the ProtectedSettings to be encrypted:

So i had some free time and an itch to understand how the different tiers really differed from each other in Azure. It’s important to understand that this is not a comparison against other cloud providers, it’s not designed to show the maximum potential in azure and the results are valid as of 05/08/2016 (most likely will be outdated in 3 months).

To elaborate on the three points above:

• I wanted to see the differences in terms of performance between the different VM size which Azure offers. I don’t want to compare them with other cloud providers as there are too many variables to considers and will be like comparing apples to oranges.
• The test i ran where identical for each VM so to show the true differences between them, while i tried to configure the test to get the max performance i can’t grantee that the results i have are the absolute maximum you can achieve nor are they the guaranteed performance you will get when you spin up a VM, especially when it comes to the network results.
• As with anything the results will probably be outdated in the next 3 months, as Azure replaces old hardware and migrates servers to new more powerful one, prime example would be the network for A series which can now achieve a lot more than the 5 – 400 mbit/s it was initially configured to have.

While the CPU and Ram tests are quite accurate, it was very tricky to get consistent network results due to it’s complexity, i guess that’s why Microsoft do not have official public results of there network and you should in no case take these results as concrete. If you think about it you have different networks over different racks, over different hardware, with Network Security group rules and so on and so on. In my case even creating different machines of the same size showed some slight variation as it most likely was on the other side of the data center. As such i have given the averages by running a lot of tests.

The way i ran the test was to create a G5 machine and install Iperf3 and run it as a server. Then i spun up a VM starting with A0 in the same network, subnet and the same NSG group. I installed Iperf3 and ran it as a client to test against the G5 machines. I ran a few tests (get the average) after which i would change the size instance to the next size up. One thing to stress this is the network speed of the internal network between virtual machines.

I did not post or test extensively the storage io as i found it to be very close to what Microsoft posted (500 iops per normal disk).

Iperf3 Settings:

• Number of threads: 30
• Packet size: 2048 KB
• Test duration: 30 seconds

Software:

• CPU,RAM Tests: Novabench
• Disk Tests: Diskspd Utility (superseding SQLIO)
• Network: Iperf3

Environment:

• VM OS: Windows Server 2012 R2
• Location: Azure West Europe
• Type:  ARM
• Network: Same Virtual network, Subnet and NSG