2005 2006 2007 2008 2009 2010 2011 2015 2016 2017 aspnet azure csharp debugging elasticsearch exceptions firefox javascriptajax linux llblgen mongodb powershell projects python security services silverlight training videos wcf wpf xag xhtmlcss

PowerShelling Azure DNS Management

DNS is one of those topics that every developer has to deal with. It's also one of those topics some of us developers get to deal with. I love DNS. It allows all manner of flexible addressing, failover, and load-balancing techniques. Understanding DNS takes you a long way in a whole lot of scenarios.

However, the difference between the fun architecture and possibilities of DNS and the implementation has often been the difference between a great sounding class syllabus and the horrible reality of a boatload of homework. Windows DNS, for example, was historically cryptic GUI gibberish. BIND was OK and has a very simple config format, but it scared most people off (you simply cannot win with people who hate GUIs and command lines).

Azure DNS lets you config via the GUI (noooooob; jk, the GUI is pretty nice), ARM templates, Azure CLI (=Linux), and PowerShell.

But, HOW do you manage the config? What format? How do you deploy?

One way is to do everything in ARM templates. Check out this example:


  "resources": [
    {
      "type": "Microsoft.Network/dnszones",
      "name": "example.org",
      "apiVersion": "2016-04-01",
      "location": "global",
      "properties": { }
    },
    {
      "type": "Microsoft.Network/dnszones/a",
      "name": "example.org/mysubdomain01",
      "apiVersion": "2016-04-01",
      "location": "global",
      "dependsOn": [
        "example.org"
      ],
      "properties": {
        "TTL": 3600,
        "ARecords": [{
            "ipv4Address": "127.0.0.1"
          }
        ]
      }
    }
  ],


There's nothing wrong with that. If you want to have a billion lines of JSON for the sake of a single DNS record, you have fun with that. Ugh. Fine. Yes, it will be idempotent, but are severe overkill for quick updates.

Because PowerShell is effectively the .NET REPL, you can write a simple PowerShell (=.NET) tool to handle any custom format you want.

The following is one way of formatting your DNS updates:


    @(
        @{
            Name="example1"
            Config=@(
            @{
                Type="CNAME"
                Value="hosting01gyfir-alpha.centralus.cloudapp.azure.com"
            })
        },
        @{
            Name= "example2"
            Config=@(
            @{
                Type="A"
                Value="127.0.0.1"
            })
        },
        @{
            Name= "mail"
            Config=@(
            @{
                Type="MX"
                Preference=10
                Exchange="mail.example.com"
            },
            @{
                Type="MX"
                Preference=20
                Exchange="mail2.example.org"
            })
        }
    )


I donno. I find that pretty simple. I'll use that when I setup something new.

Here's my function (with call) to make that work:


    function deployDns { param([Parameter(Mandatory=$true)]$rg, [Parameter(Mandatory=$true)]$zonename, [Parameter(Mandatory=$true)] $ttl, [Parameter(Mandatory=$true)]$records, [bool]$testOnly)
        ($records).foreach({
            $name = $_.Name
            Write-Host "Name: $name"
            $dnsrecords = @()
    
            ($_.Config).foreach({
                $config = $_
                $type = $config.Type
                switch($type) {
                    "CNAME" {
                        Write-Host ("`tCNAME: {0}" -f $config.Value)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Cname $config.Value
                    }
                    "MX" {
                        Write-Host ("`tPreference: {0}" -f $config.Preference)
                        Write-Host ("`tExchange: {0}" -f $config.Exchange)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Preference $config.Preference -Exchange $config.Exchange
                    }
                    "A" {
                        Write-Host ("`tPreference: {0}" -f $config)
                        Write-Host ("`tExchange: {0}" -f $config.Ipv4Address)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Ipv4Address $config.Value
                    }
                    "AAAA" {
                        Write-Host ("`tIpv6Address: {0}" -f $config.Ipv6Address)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Ipv6Address $config.Value
                    }
                    "NS" {
                        Write-Host ("`tNS: {0}" -f $config.Value)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Nsdname $config.Value
                    }
                    "PTR" {
                        Write-Host ("`tPtrdname: {0}" -f $config.Value)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Ptrdname $config.Value
                    }
                    "TXT" {
                        Write-Host ("`tPtrdname: {0}" -f $config.Value)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Value $config.Value
                    }
                }
            })
            Write-Host
            Write-Host "Records:"
            Write-Host $dnsrecords
            if(!$testOnly) {
                New-AzureRmDnsRecordSet -ResourceGroupName $rg -ZoneName $zonename -RecordType $type -Name $name -Ttl $ttl -DnsRecords @dnsrecords
            }
            Write-Host
            Write-Host
        })
    }
    
    deployDns -testOnly $true -rg 'davidbetz01' -zonename "davidbetz.net" -ttl 3600 -records @(
        @{
            Name="example1"
            Config=@(
            @{
                Type="CNAME"
                Value="hosting01gyfir-alpha.centralus.cloudapp.azure.com"
            })
        },
        @{
            Name= "example2"
            Config=@(
            @{
                Type="A"
                Value="127.0.0.1"
            })
        },
        @{
            Name= "mail"
            Config=@(
            @{
                Type="MX"
                Preference=10
                Exchange="mail.example.com"
            },
            @{
                Type="MX"
                Preference=20
                Exchange="mail2.example.org"
            })
        }
    )


Is this insane? Probably. That's what I'm known for. ARM templates might be smarter given their idempotent nature, and I've found myself using the GUI now and again.

For now, just keep in mind that PowerShell lets you me ultra flexible with your Azure configuration, not just Azure DNS.

Allowing access to your Azure VM

You created your VM, you installed and configured all services, and your firewalld/iptables is set correctly. Your nmap tests are even working between systems.

But, you can't access your services external to Azure?

You probably didn't enable access in Azure. You need to allow specific ports in your Azure Network Security Group.

In terms of your Azure objects, your VM uses a NIC, your NIC uses an NSG.

Using PowerShell

Using PowerShell, you can do something like this:

    
    $rg = 'hosting01'
    
    $nsg = Get-AzureRmNetworkSecurityGroup -ResourceGroupName $rg -Name "$rg-nsg-alpha"
    
    $maximum = ($nsg.SecurityRules | measure -Property priority -Maximum).Maximum + 100
    $httpRule = New-AzureRmNetworkSecurityRuleConfig -Name "http" -Protocol Tcp -SourceAddressPrefix * -DestinationAddressPrefix * -SourcePortRange * -DestinationPortRange 80 -Priority $maximum -Description "HTTP" -Direction Inbound -Access Allow
    $nsg.SecurityRules.Add($httpRule)
    
    $maximum = ($nsg.SecurityRules | measure -Property priority -Maximum).Maximum + 100
    $httpsRule = New-AzureRmNetworkSecurityRuleConfig -Name "https" -Protocol Tcp -SourceAddressPrefix * -DestinationAddressPrefix * -SourcePortRange * -DestinationPortRange 443 -Priority $maximum -Description "SSL" -Direction Inbound -Access Allow
    $nsg.SecurityRules.Add($httpsRule)
    
    Set-AzureRmNetworkSecurityGroup -NetworkSecurityGroup $nsg


Use an ARM Template

Or you can just fix your initial ARM template by adding the resource:

See the https://linux.azure.david.betz.space/_/python-uwsgi-nginx on https://linux.azure.david.betz.space for a fuller example.


    {
        "comments": "",
        "type": "Microsoft.Network/networkSecurityGroups",
        "name": "nsg-alpha",
        "apiVersion": "2017-03-01",
        "location": "[resourceGroup().location]",
        "properties": {
            "securityRules": [
                {
                    "name": "default-allow-ssh",
                    "properties": {
                        "protocol": "Tcp",
                        "sourcePortRange": "*",
                        "destinationPortRange": "22",
                        "sourceAddressPrefix": "*",
                        "destinationAddressPrefix": "*",
                        "access": "Allow",
                        "priority": 1000,
                        "direction": "Inbound"
                    }
                },
                {
                    "name": "http",
                    "properties": {
                        "protocol": "Tcp",
                        "sourcePortRange": "*",
                        "destinationPortRange": "80",
                        "sourceAddressPrefix": "*",
                        "destinationAddressPrefix": "*",
                        "access": "Allow",
                        "priority": 1100,
                        "direction": "Inbound"
                    }
                },
                {
                    "name": "https",
                    "properties": {
                        "protocol": "Tcp",
                        "sourcePortRange": "*",
                        "destinationPortRange": "443",
                        "sourceAddressPrefix": "*",
                        "destinationAddressPrefix": "*",
                        "access": "Allow",
                        "priority": 1200,
                        "direction": "Inbound"
                    }
                }
            ]
        },
        "resources": [],
        "dependsOn": []
    }


Add this to your NICs (Microsoft.Network/networkInterfaces) properties:

    
    "type": "Microsoft.Network/networkInterfaces",
    "properties": {
        "networkSecurityGroup": {
            "id": "[resourceId('Microsoft.Network/networkSecurityGroups', concat(variables('nsg-prefix'), variables('names')[0]))]"
        }
    }


...and dependsOn section:


    
    "dependsOn": [
        "[resourceId('Microsoft.Network/networkSecurityGroups', concat(variables('nsg-prefix'), variables('names')[0]))]"
    ]


Using Azure DNS

Though you might be used to doing DNS via bind, there's nothing specific about bind that defines DNS. You can do DNS anywhere; all DNS details are blackboxed.

As such, Azure DNS is perfectly compatible with being a super Linux geek.

You can review the docs for Azure DNS on your own, but I'd like to provide a few "recipes".

Everything that follows assumes that you've done the prerequisite work of getting an active Azure account, creating your resource group (New-AzureRmResourceGroup) and creating a new DNS zone (New-AzureRmDnsZone).


New-AzureRmResourceGroup -Name spartan01
New-AzureRmDnsZone -ResourceGroupName spartan01 -Name "example.com"

In moving to Azure DNS, you have a few things that you need to do. These have NOTHING to do with Azure -- they're just facts of DNS.

PowerShell will be used for the most part, but the Azure CLI syntax will be provided where it's particularly interesting.

E-Mail (MX Records)

First, you need to realize that DNS affects your e-mail. This should be obvious since your e-mail has a domain name right in it. So, you need to make sure your MX records are legit.

Here's a block of PowerShell (UpdateAzureDNSMX.ps1) you should keep around:


    New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType A -Name "mail" -Ttl 3600 -DnsRecords (New-AzureRmDnsRecordConfig -Ipv4Address '203.0.113.1')
    
    New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType MX -Name "mail" -Ttl 3600 -DnsRecords @(
        (New-AzureRmDnsRecordConfig -Preference 10 -Exchange smtp.example.com),
        (New-AzureRmDnsRecordConfig -Preference 20 -Exchange mailstore1.example.com)
    )

This creates a record set, then adds the specific config.

Single IP Address

Let's say you have a bunch of sites and want a single server to handle them all. Let's also assume that you only have one IP address.

You should look into using WebApps, but let's pretend you absolutely need a VM for some wild reason.

What to do?

Simple: CNAME it.

After creating your shiny new VM (perhaps running ./create simple hosting01 from https://linux.azure.david.betz.space), you need to get the systems IP address:

Azure CLI

[dbetz@callisto ~]$ az network public-ip list --query "[?dnsSettings.domainNameLabel!=null]|[?starts_with(dnsSettings.domainNameLabel,'hosting01')].{ dns: dnsSettings.fqdn  }"           
[
  {
    "dns": "hosting01figrg-alpha.centralus.cloudapp.azure.com"
  }
]

PowerShell


    (Get-AzureRmPublicIpAddress -ResourceGroupName hosting01 -Name hosting01-ip-alpha).DnsSettings.Fqdn

Let's use that Azure domain name to create our CNAME records for our domains:


    function create { param($name)
        New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType CNAME -Name $name -Ttl 3600 -DnsRecords (New-AzureRmDnsRecordConfig -Cname "hosting01figrg-alpha.centralus.cloudapp.azure.com")
    }
    create "subdomain01" 
    create "subdomain02" 
    create "subdomain03" 
    create "subdomain04" 

For the sake of this example, all domains will be as subdomains. Doing full, separate domains is the same, but the examples would be longer... and less fun. It's the same idea, though.

If you already have the records (e.g. you move to another VM later), you update:

    
    function update { param($name)
        $rs = Get-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com"  -RecordType CNAME -Name $name
        $rs.Records[0] = (New-AzureRmDnsRecordConfig -Cname "hosting01figrg-alpha.centralus.cloudapp.azure.com")
        Set-AzureRmDnsRecordSet -RecordSet $rs
    }
    update "subdomain01" 
    update "subdomain02" 
    update "subdomain03" 
    update "subdomain04" 

Now you have your domains all pointing to the same place.

Stopping here doesn't help you. You still have to figure out how to get your server to handle the multiple sites! Though it's not directly Azure related, I find partial examples to be distasteful and a sign of extreme ignorance on the part of an author.

So, what do you do on the server?

Let's assume you're reading this after 2016 (I'm writing this in 2017, sooooo...), therefore you're not using Apache. You're using Nginx:

server {
    listen 80;

    server_name mysubdomain01.example.com;
    
    return 301 https://mysubdomain01.example.com$request_uri;
}

server {
    listen 443 ssl http2;

    server_name mysubdomain01.example.com;

    ssl_certificate /etc/letsencrypt/live/mysubdomain01.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysubdomain01.example.com/privkey.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

    ssl_prefer_server_ciphers on;

    ssl_dhparam /srv/_cert/dhparam.pem;

    location / {
        add_header Strict-Transport-Security max-age=15552000;
        proxy_pass http://127.0.0.1:8081;        
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Port 443;
        proxy_set_header Host mysubdomain01.example.com;
    }
}

I'm not into trivial examples, so the full SSL example is provided, with the letsencrypt paths.

The point is this:

  • listen on the port without the IP
  • server the server_name to your full domain name

Done.

Well, no, we're not done to my satisfaction. You still need some ability to test this. So, here's a quick Node.js application you can use to listen on port 80. Just curl from somewhere else and see if it works.

curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -
yum -y install nodejs

cd
cat > server.js << EOF
http = require('http');
port = parseInt(process.argv[2]);
server = http.createServer( function(req, res) {
    res.writeHead(200, {'Content-Type': 'text/html'});
    res.end(req.method + ' server ' + port);
});
host = '$PUBLIC_IP';
server.listen(port, host);
EOF

node server.js 8081 &

Azure CDN

When using assets (e.g. png, css) on your website, you'll want to be sure to serve them via an Azure CDN.

I tend to avoid the term "blob" as that's a binary entity -- the moment you start talking about "SVG blobs" or "CSS blobs", you lose any connection to reality. They're assets.

If your CDNS provider (Azure uses both Akamai and Verizon) supports SSL for custom domains, then great. If not, you're stuck with the mycdnendpoint.azurewebsites.net address. You can consider HTTP obsolete. Use SSL or do not host anything.

Assuming your CDN provider supports SSL for custom domains, you have to tell it about your custom domain.

It requires that you prove ownership by creating a cdnverify CNAME entry. Most of the docs are deeply cryptic on this (honestly, RTFM means FU; it's rarely helpful), so I'll make it simple:

if you want to use cdn01.mydomain.net instead of http://mycdnendpoint.azurewebsites.net, you can do this in two ways (choose one):

  • create a cdn01 CNAME pointing to mycdnendpoint.azurewebsites.net

New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType CNAME -Name 'cd01' -Ttl 3600 -DnsRecords (New-AzureRmDnsRecordConfig -Cname "mycdnendpoint.azureedge.net")


OR

  • create a cdnverify.cdn01 CNAME pointing to cdnverify.mycdnendpoint.azurewebsites.net.

New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType CNAME -Name 'cdnverify.cd01' -Ttl 3600 -DnsRecords (New-AzureRmDnsRecordConfig -Cname "cdnverify.mycdnendpoint.azureedge.net")


NS Records

Once your DNS is setup and ready for production, you need to tell your registrar about it.

You can't guess what these records are. You just ask Azure about them:


    (Get-AzureRmDnsZone -ResourceGroupName spartan01 -Name "example.com").NameServers

Here's a preview of something from namecheap.com (use whatever you want... as long as it's not GoDaddy)

ARM Components

Using Azure App Services with Node.js

Often we hear about how Azure is "Microsoft" whereas other cloud providers aren't. In the most obvious sense, they're right -- Microsoft owns it. However, when you look closer what they actually mean is that Azure is Microsoft-only and Google/AWS are open to other programming models.

This is ridiculous and only said by people gulping down electrolyte-loaded propaganda by the water cooler. In reality, there's nothing proprietary or Microsoft-only about Azure as a whole. It's a nonsensical bias to say otherwise. Azure is platform agnostic. You don't need a VM (=IAAS) to do your Node (or Python development). Absolutely zero hacks are required to put your Node application in Azure App Services (=PAAS).

It reminds me of when HTML5 became popular: there we still Flash-zealots pushing their "LOL browsers can't do animation nonsense LOL". They forgot to leave their echo chamber before attempting to entering reality.

Software engineers steeped in Microsoft technologies for over a decade understand that you must make a distinction between Ballmer and non-Ballmer Microsoft. To give an extreme contrast: the former is VB6, the latter is .NET Core. You can see the transition from Ballmer-nonsense to non-Ballmer-sanity since around .NET 4 (especially in the adoption of ASP.NET MVC as an open-source tool). Ballmer is Silverlight, non-Ballmer is adoption-of-HTML5. You can go on down the line.

In the end: Ballmer-Microsoft is Microsoft-as-evil-empire. Today's Microsoft is fully amiable toward Linux; they also rely on Github for many SDKs and for just about all Azure documentation. It's a different beast.

You need to examine Azure through this Ballmer / non-Ballmer paradigm. Put concretely: Windows Azure (and Azure ASM) was Ballmer whereas Microsoft Azure (and Azure ARM) is non-Ballmer. Much of the "LOL Azure is Microsoft-only LOL" nonsense comes from confusion about the transition between Azure "versions". Much of this is also Microsoft's fault: just about all the books out there are completely obsolete! The official-book for the 70-532 exam will absolutely guarantee that you fail the exam.

For the topic at hand, we shouldn't look at Azure in an ad hoc manner, but in the context of it's intimately related technologies. Specifically we need to look at the development of IIS as it passed from a Ballmer to a non-Ballmer implementation.

Working with IIS

My life with IIS started around the IIS3 era. I still remember taking the IIS3 exam as an elective (with the TCP/IP exam) for my NT4 MCSE. Thus, I've seen the various large upgrades and incremental updates over a good stretch of time.

The upgrade from IIS6 to IIS7 was easily the largest IIS upgrade; it laid the groundwork for eventually stripping out the last vestige of Ballmerisms via its flexible APIs. Till IIS7, the biggest upgrade was a silly configuration system update (=IIS4 metabase). The IIS7 upgrade consisted of a systematic, paradigmatic shift. It was the "classical" to the "integrated" pipeline upgrade. The upgrade to so deep, that you literally had to update your applications to add IIS7 support. After a while, all development was IIS7-first with IIS6 backwards compatibility added subsequently.

In practice, this classic -> integrated upgrade meant three things: First, instead of relying on the external ASP.NET ISAPI IIS plug-in, ASP.NET processing was integrated into IIS. No more interop. This made ASP.NET development more natural. It also gave .NET access to core extensibility functionality in IIS. You didn't need to whip out C++ for server extensibility. Second, if you had existing C++ functionality, you had easier access to IIS functionality with the new native IIS API. This second point is critical, because we see that the IIS7 upgrade wasn't just about .NET. Third, web.config was no longer about ASP.NET, but about IIS itself. This point is huge and points to the fact that the web.config format controls all over IIS7+, as seen in the global applicationHost.config file.

IIS6 used the rediculous ISAPI nonsense to do just about everything, including call ASP.NET. The .aspx extension was simply mapped to aspnet_isapi.dll. This wasn't removed from IIS7; it was just separated and called "classic" mode.

In this a IIS7 world, this meant that you literally had to add handler / module support for both IIS6 and IIS7 (more accurately, the classical and integrated models).

Furthermore, the low-level ASP.NET pipeline APIs were also affected. For my deeply low-level Themelia framework, I had to make checks between completely different pipelines. See the following snippet from my CoreModule (a typical module implementing the System.Web.IHttpModule interface):

View Themelia at Themelia Pro. View Themelia source at Themelia on Github

CoreModule.cs:

    
    if (HttpRuntime.UsingIntegratedPipeline)
    {
        httpApplication.PostResolveRequestCache += OnProcessRoute;
        httpApplication.PostMapRequestHandler += OnSetHandler;
    }
    else
    {
        httpApplication.PostMapRequestHandler += OnProcessRoute;
        httpApplication.PostMapRequestHandler += OnSetHandler;
    }

Reference: CoreModule.cs

The installation was also different between IIS6 ("classical") and IIS7 ("integrated"):

For IIS6, I would add the module to system.web:


    <system.web>
        <httpModules>
            <add name="Themelia" type="Themelia.Web.CoreModule, Themelia.Web"/>
        </httpModules>
    </system.web>

For IIS7, I would add the module to system.webServer:


    <system.webServer>
        <modules>
            <remove name="Session"/>
            <add name="Session" type="System.Web.SessionState.SessionStateModule" preCondition=""/>
            <add name="Themelia" type="Themelia.Web.CoreModule, Themelia.Web"/>
        </modules>
    </system.webServer>

There is also the much more popular concept of a handler. My framework was meant to be a full IIS6-era platform takeover, so I used a more greedy module, but if you're only doing specific framework development, handlers are your choice. ASP.NET MVC, for example, uses handler.

ASP.NET MVC is actually an excellent example. Assuming ASP.NET was properly installed, for IIS7, they were able to take advantage of the fact that IIS7 processed everything (e.g. /contact) as .NET (though you still needed runAllManagedModulesForAllRequests enabled to disable that slight perf boost). For IIS6, because it had to know when to call the ASP.NET ISAPI filter, you had to add a wildcard handling to get the ISAPI filter handle extenstionless paths (again, e.g. /contact).


    <system.webServer>
        <handlers>
            <add verb="*" path=".png" name="WatermarkHandler" type="WatermarkHandler"/>
        </handlers>
    </system.webServer>

Handlers and modules are still the standard was of tapping into the stream of raw power.

.NET is powerful. C# even has an unmanaged mode where you can crack open the covers (via unsafe mode) to do direct *pointer &manipulation. That said, the upgrade to IIS7 wasn't just about .NET; the upgrade provided a native IIS API as well.

Thus we enter the realm of C/C++ modules: Develop a Native C\C++ Module for IIS 7.0

By removing the ISAPI barrier and providing a clean, native IIS API C++ developers could more easily connect existing C++ functionality to IIS. It also made ASP.NET C++ code more expressive; familiar web terms like HttpContext, IHttpResponse, and BeginRequest (and other events) are all over IIS C++ code. No more DWORD WINAPI HttpExtensionProc(EXTENSION_CONTROL_BLOCK *pECB) nonsense.

Seriously. Review the C++ ISAPI docs. They're insane. 1990s Microsoft C++ was the worst code ever written. It's just plain satanic.

Consider the following IIS7-esque native C++ method:


    HRESULT        
    __stdcall        
    RegisterModule(        
        DWORD                           dwServerVersion,    
        IHttpModuleRegistrationInfo *   pModuleInfo,
        IHttpServer *                   pHttpServer            
    )
    {
    }

That's exactly how you register your native modules in IIS7. That's not too terribly evil. You can see that it's registering a module, and bringing in points to core IIS entities.

This is also exactly how IIS handles Node hosting in Azure; it uses the iisnode module. You can see RegisterModule in main.cpp in iisnode:

https://github.com/tjanczuk/iisnode/blob/master/src/iisnode/main.cpp

If you review the following code from CProtocolBridge.cpp in iisnode, you'll see familiar things like IHttpContext and IHttpResponse:

https://github.com/tjanczuk/iisnode/blob/master/src/iisnode/cprotocolbridge.cpp

It's clean interface programming.

Using iisnode

IIS handles most of it's config with your applications web.config. While there are a few global config files, you get tremendous control with your own config.

Hosting a Node application in Azure is as simple as deploying an Azure Web App with a properly configured web.config.

You can following along with the following activities by deploying the following repo -> https://github.com/davidbetz/template-azure-node-api.

Per the previous explanation of IIS modules, you can see from the following web.config that iisnode is installed just as we would install our own handlers and modules. There are no hacks whatsoever.

    
    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
      <system.webServer>
        <!-- leave false, you enable support in Azure -->
        <webSocket enabled="false" />
        <handlers>
          <add name="iisnode" path="server.js" verb="*" modules="iisnode"/>
        </handlers>
        <rewrite>
          <rules>
            <rule name="StaticContent">
              <action type="Rewrite" url="content{REQUEST_URI}"/>
            </rule>
            <rule name="DynamicContent">
              <conditions>
                <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="True"/>
              </conditions>
              <action type="Rewrite" url="server.js"/>
            </rule>
            <rule name="Redirect to https" stopProcessing="true">
              <match url="(.*)" />
              <conditions>
                <add input="{HTTPS}" pattern="off" ignoreCase="true" />
              </conditions>
              <action type="Redirect" url="https://{HTTP_HOST}{REQUEST_URI}" redirectType="Permanent" appendQueryString="false" />
            </rule>
          </rules>
        </rewrite>
        <security>
          <requestFiltering>
            <hiddenSegments>
              <remove segment="bin"/>
            </hiddenSegments>
          </requestFiltering>
        </security>
        <httpErrors existingResponse="PassThrough" />
      </system.webServer>
    </configuration>

The following section listens for any requests for all verbs accessing server.js and has iisnode process them:


    <handlers>
        <add name="iisnode" path="server.js" verb="*" modules="iisnode"/>
    </handlers>

The following rewrite rule sends all traffic to server.js:


    <rule name="DynamicContent">
        <conditions>
            <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="True"/>
        </conditions>
        <action type="Rewrite" url="server.js"/>
    </rule>

The following doesn't have anything directly to with iisnode; it excludes the content folder from iisnode processing:


    <rule name="StaticContent">
        <action type="Rewrite" url="content{REQUEST_URI}"/>
    </rule>

I find putting static files on your web server to be naive, but if you really don't want to use the Azure CDN, that is how you host static content.

The following merely redirects HTTP to HTTPS:


    <rule name="HTTP to HTTPS redirect" stopProcessing="true">
        <match url="(.*)" />
        <conditions>
            <add input="{HTTPS}" pattern="off" ignoreCase="true" />
        </conditions>
        <action type="Redirect" url="https://{HTTP_HOST}/{REQUEST_URI}" redirectType="Permanent" />
    </rule>

Breaking iisnode

To prove that Node hosting is actually this basic, let's break it, then fix it.

First, let's see this work:

https://node38eb089b-app-alpha.azurewebsites.net/api/samples

it works

That's the expected output from the application.

Now, let's go to web.config and break it:


  <rule name="DynamicContent">
    <conditions>
      <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="True"/>
    </conditions>
    <action type="Rewrite" url="server.js"/>
  </rule>

Change that Rewrite from server.js to server2.js:

Save. Refresh browser.

404

Nope.

Go to Kudu. This is the .scm. URL. In this case it's the following:

https://node3638972b-app-alpha.scm.azurewebsites.net/

kudu

Rename server.js to server2.js:

rename server.js to server2.js

Refresh again.

rawcodeoutput

mmmk. Raw output.

The rewrite is telling everything to go to server.js, but nothing is processing it, so it just sends the file back.

This is exactly like accessing an old .aspx page and getting the raw ASP.NET webform code, because you forgot to install ASP.NET (and somehow managed to allow access to .aspx).

Now, let's fix this by telling our IIS module to process server2.js:


    <handlers>
      <add name="iisnode" path="server2.js" verb="*" modules="iisnode"/>
    </handlers>

Refresh and it's all well again:

it works

App Services and App Service Plan Mechanics

An explanation of Azure web apps using any web platform isn't complete without reviewing the mechanic of Azure web apps.

To begin, let's clarify a few Azure terms:

Azure App Service Plans are effectively managed VMs. You can scale these up and out. That is, you can turn an S1 into an S2 to double the RAM or you can turn a single instance into four. Because of this later ability, App Service Plans are also known as server farms. In fact, when developing ARM templates, the type you use to deploy an App Service Plan is Microsoft.Web/serverfarms.

You do not deploy a series of plans to create a farm. Plans are farms. Plans with a size of 1 is just a farm with 1 instance. You are always dealing with herds, you are never dealing with pets. You scale your farm out, you scale all those instances up.

Azure Web Apps are also known as Web Sites and App Services. You deploy these, you back these up, and you add SSL to these. These are similar to IIS virtual applications. When developing ARM templates, the type is Microsoft.Web/sites.

You do need to remember the various synonyms for each; you will see them all.

Given this distinction and given the fact that a VM can have multiple IIS applications, you can imagine that you can host multiple Azure Web Apps on a single App Service Plan. This is true. You do NOT deploy a plan every time you deploy a site. You plan your CPU/RAM usage capacity ahead of time and deploy a full solution at once.

To visualize the App Service / App Service Plan distinction, review the following image.

Here I've provide information for three services over two service plans. The first two services share a service plan, the third service is on a different plan.

Notice that the services with the same service plan have the same machine ID and instance ID, but their IIS specifics are different. The third service plan has a different machine ID altogether.

What's so special about the types of Web Apps?

If this is all just the same IIS, what's with the various Node-specific web app types?

web app types

The answer is simple: they exist solely to confuse you.

Fine. Whatever. The different types are hello-world templates, but you're going to overwrite them via deployment anyway.

You can literally deploy a Node.js web app, then deploy and ASP.NET site on it. It's just IIS. The website deployment will overwrite the web.config with its own.

Given the previous explanations of IIS handler/modules, iisnode-as-module, and the service/plan distinction, you can see that there's no magic. There's nothing Microsoft-only about any of this.

You can always use the normal "Web App" one and be done with it.

Single App Solutions

My websites are generally ASP.NET or Python/Django, but my APIs are always Node (ever since someone at Microsoft with an IQ of a Pennsylvania speed limit decided to deprecate Web API and rebuild it as "MVC" in ASP.NET Core). There was a time when my APIs and my website required separate... just about everything. Now adays I use nginx as a single point of contact to serve traffic from various internal sources: one source to handle the website as a whole (either http://127.0.0.1:XXXX or a Linux socket) and another to handle /api. This lets me use a single domain (therefore a single SSL cert) for my solution.

In Azure, this functionality is provided by Application Gateway.

Think back through all the mechanics we've covered so far: IIS can handle .NET and supports modules. iisnode is a module. IIS uses rewriting to send everything to server.js. iisnode handles all traffic sent to server.js.

Let's mix this up: instead of rewriting everything to server.js, let's only rewrite the /api branch of our URL.

To make this example a bit spicier, let's deploy an ASP.NET MVC application to our App Service, then send /api to Node.

To do this, go to the App Service (not the App Service Plan!), then select Deployment Options on the left.

external

In Choose source, select External Repository and put in the following:

https://github.com/Azure-Samples/app-service-web-dotnet-get-started

external

A few minutes later, load the application normally. You'll see the "ASP.NET is a free web framework for..." propaganda.

Now go back into Kudu and the PowerShell Debug Console (explained earlier).

We need to do three things:

  • add our server.js
  • install express
  • tell web.config about server.js

To add server.js, go to site/wwwroot and type the following:

touch server.js

This will create the file. Edit the file and paste in server from the following:

https://github.com/davidbetz/template-azure-node-api/blob/master/server.js

Next we need to install express to handle the API processing. H

For the sake of a demo, type the following:

npm install express

Done.

For the sake of your long-term sanity, create package.json (same way you created server.js), edit in contents, save, then run:

npm install

See sample package.json at:

https://raw.githubusercontent.com/davidbetz/template-azure-node-api/master/package.json

Finally, edit web.config.

You need the splice in the following config in the system:


  <system.webServer>
    <rewrite>
      <rules>
        <rule name="DynamicContent">
          <match url="^api/(.*)" />
          <action type="Rewrite" url="server.js"/>
        </rule>
      </rules>
    </rewrite>
    <handlers>
      <add name="iisnode" path="server.js" verb="*" modules="iisnode"/>
    </handlers>
  </system.webServer>

Upon saving, access the web app root and /api/samples. Click around the web app to prove to yourself that it's not just a static page.

asp.net and node together

You have ASP.NET and Node.js in the same Azure web app. According to a lot of the FUD out there, this shouldn't be possible.

In addition to hosting your SPA and your APIs in the same place, you also don't need to play with CORS nonsense. You also don't need an Application Gateway (=Microsoft's nginx) to do the same thing

Scenario: Deploy Multi-Region Elasticsearch Cluster on Azure

Previously, I wrote about my proposal for modular ARM templates. Here I'm going to go through the scenario of deploying a full, multi-location Elasticsearch node cluster with a single public endpoint and pretty DNS.

This is a sequel to Developing Azure Modular ARM Templates. Study that first. You need the Deploy ARM Template PowerShell at the bottom to complete this deployment.

Let's begin by reviewing the solution:

This solution is at https://github.com/davidbetz/azure-elasticsearch-nodes-scenario

solution files

Here we see:

  • A 3-phase Azure deployment
  • 3 PowerShell files
  • A VM setup script (install.sh)
  • A script that install.sh will run (create_data_generation_setup.sh)
  • A file that create_data_generation_setup.sh will indirectly use (hamlet.py)

The modular deployment phases are:

  • Setup of general resources (storage, vnet, IP, NIC, NSG)
  • Setup of VNet gateways
  • Setup of VMs

The first phase exists to lay out the general components. This is a very quick phase.

The second phase creates the gateways.

After the second phase, we will create the VNet connections with PowerShell

The third phase will create the VMs

Then we will generate sample data and test the ES cluster.

Then we will create an Azure Traffic Manager and test.

Finally we will add a pretty name to the traffic manager with Azure DNS.

Let's do this...

Creating Storage for Deployment

The first thing we're going to do is create a storage account for deployment files. In various examples online, you see this in Github. We're not going to do that. Here we create a storage account that future phases will reference. You only need one... ever. Here I'm creating one just for this specific example deployment.


    $uniquifier = $([guid]::NewGuid().tostring().substring(0, 8))
    $rg = "esnodes$uniquifier"
    _createdeploymentAccount -rg $rg -uniquifier $uniquifier

Reference the Deploy ARM Template PowerShell file at the end of Developing Azure Modular ARM Templates for the above and later code.

Output

VERBOSE: Performing the operation "Replacing resource group ..." on target "".
VERBOSE: 11:01:28 PM - Created resource group 'esnodesfbdac204' in location 'centralus'


ResourceGroupName : esnodesfbdac204
Location          : centralus
ProvisioningState : Succeeded
Tags              : 
ResourceId        : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204


ResourceGroupName      : esnodesfbdac204
StorageAccountName     : filesfbdac204
Id                     : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Storage/storageAccounts/filesfbdac204
Location               : centralus
Sku                    : Microsoft.Azure.Management.Storage.Models.Sku
Kind                   : BlobStorage
Encryption             : 
AccessTier             : Hot
CreationTime           : 7/22/2017 4:01:29 AM
CustomDomain           : 
Identity               : 
LastGeoFailoverTime    : 
PrimaryEndpoints       : Microsoft.Azure.Management.Storage.Models.Endpoints
PrimaryLocation        : centralus
ProvisioningState      : Succeeded
SecondaryEndpoints     : 
SecondaryLocation      : 
StatusOfPrimary        : Available
StatusOfSecondary      : 
Tags                   : {}
EnableHttpsTrafficOnly : False
Context                : Microsoft.WindowsAzure.Commands.Common.Storage.LazyAzureStorageContext
ExtendedProperties     : {}


CloudBlobContainer : Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer
Permission         : Microsoft.WindowsAzure.Storage.Blob.BlobContainerPermissions
PublicAccess       : Blob
LastModified       : 7/22/2017 4:02:00 AM +00:00
ContinuationToken  : 
Context            : Microsoft.WindowsAzure.Commands.Storage.AzureStorageContext
Name               : support

Phase 1 Deployment

Now we're ready for phase 1 of deployment. This phase is quick and easy. It simply creates the basic components that future phases will use.

Part of what phase 1 will deploy is a virtual network per region that we're requesting. In this example, we have "central us", "west us", and "east us". We need a virtual network in each.

But, to make this work we have to remember our IP addressing:

Very short IP address review

Given a 10.x.x.x network, and a 0.y.y.y mask, 10 is your network and the x.x.x is your host area.

Given a 10.1.x.x network, and a 0.0.y.y mask, 10.1 is your network and x.x is your mask.

The concept of subnetting is relative to the network and only shows up in discussions of larger scale networks and supernetting. Tell the pendantic sysadmins to take a hike when they they to confuse you with over emphasising the network vs. subnetwork aspects. This is a semantic concept, not a technical one. That is, it relates to the design, not the bits themselves.

The virtual networks in our modular deployment uses the following addressSpace:


    "addressSpace": {
        "addressPrefixes": [
            "[concat('10.', mul(16, add(copyIndex(), 1)), '.0.0/12')]"
        ]
    },

We see that our networks follow a 10.16*n+1.0.0/12 pattern.

This takes n to generate networks: n=0 => 10.16.0.0, n=1 => 10.32.0.0, and n=2 => 10.48.0.0.

Azure allows you to split your networks up into subnets as well. This is great for organization. Not only that, when you specify a NIC, you put it on a subnet. So, let's look at our subnet configuration:

    
    "subnets": [
        {
            "name": "subnet01",
            "properties": {
                "addressPrefix": "[concat('10.', add(mul(16, add(copyIndex(), 1)), 1), '.0.0/16')]"
            }
        },
        {
            "name": "subnet02",
            "properties": {
                "addressPrefix": "[concat('10.', add(mul(16, add(copyIndex(), 1)), 2), '.0.0/16')]"
            }
        },
        {
            "name": "GatewaySubnet",
            "properties": {
                "addressPrefix": "[concat('10.', mul(16, add(copyIndex(), 1)), '.0.', 16,'/28')]"
            }
        }
    ]

The NICs for our VMs will be on subnet01. We will not be using subnet02, but I always include it for future experiments and as an example of further subnetting.

GatewaySubnet is special and is used only by the VPN gateways. Don't mess with that.

Zooming into subnet01, we see a 10.(16*n+1)+1.0.0/16 pattern. It's basically the network + 1 with the next four bits defining the subnet (in our case the subnet is the network; it's only a subnet from the perspective of the network, but we're not viewing it from that perspective).

This takes n to generate networks: n=0 => 10.17.0.0, n=1 => 10.33.0.0, and n=2 => 10.49.0.0.

End of mini-lesson.

Now to deploy phase 1...

Output

(filtering for phase 1)
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\networkInterfaces\nic-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\networkSecurityGroups\nsg-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\publicIPAddresses\pip-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\virtualNetworks\vnet-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\storage\storageAccounts\storage-copyIndex.json...
(excluding \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\publicIPAddresses\2.pip-gateway-copyIndex.json)
(excluding \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\virtualNetworkGateways\2.gateway-copyIndex.json)
(excluding \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\compute\virtualMachines\3.vm-copyIndex.json)
(excluding \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\compute\virtualMachines\extensions\3.script.json)
------------------------------------

------------------------------------
Creating \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\deploy\07212017-110346.1...
Deploying template \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\deploy\azuredeploy-generated.json
VERBOSE: Performing the operation "Creating Deployment" on target "esnodesfbdac204".
VERBOSE: 07:03:49 PM - Template is valid.
VERBOSE: 07:03:51 PM - Create template deployment 'elasticsearch-secure-nodes07212017-110346'
VERBOSE: 07:03:51 PM - Checking deployment status in 5 seconds
VERBOSE: 07:03:56 PM - Checking deployment status in 5 seconds
VERBOSE: 07:04:01 PM - Checking deployment status in 5 seconds
VERBOSE: 07:04:06 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204alpha' provisioning status is running
VERBOSE: 07:04:06 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-gamma' provisioning status is running
VERBOSE: 07:04:06 PM - Checking deployment status in 5 seconds
VERBOSE: 07:04:12 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gamma' provisioning status is succeeded
VERBOSE: 07:04:12 PM - Resource Microsoft.Network/publicIPAddresses 'pip-beta' provisioning status is succeeded
VERBOSE: 07:04:12 PM - Resource Microsoft.Network/publicIPAddresses 'pip-alpha' provisioning status is succeeded
VERBOSE: 07:04:12 PM - Resource Microsoft.Network/virtualNetworks 'vnet-gamma' provisioning status is succeeded
VERBOSE: 07:04:12 PM - Resource Microsoft.Network/virtualNetworks 'vnet-alpha' provisioning status is succeeded
VERBOSE: 07:04:12 PM - Resource Microsoft.Network/virtualNetworks 'vnet-beta' provisioning status is succeeded
VERBOSE: 07:04:12 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-alpha' provisioning status is running
VERBOSE: 07:04:12 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204gamma' provisioning status is running
VERBOSE: 07:04:12 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-beta' provisioning status is running
VERBOSE: 07:04:12 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204beta' provisioning status is running
VERBOSE: 07:04:12 PM - Checking deployment status in 5 seconds
VERBOSE: 07:04:17 PM - Checking deployment status in 5 seconds
VERBOSE: 07:04:22 PM - Resource Microsoft.Network/networkInterfaces 'nic-alpha' provisioning status is succeeded
VERBOSE: 07:04:22 PM - Resource Microsoft.Network/networkInterfaces 'nic-gamma' provisioning status is succeeded
VERBOSE: 07:04:22 PM - Resource Microsoft.Network/networkInterfaces 'nic-beta' provisioning status is succeeded
VERBOSE: 07:04:22 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-alpha' provisioning status is succeeded
VERBOSE: 07:04:22 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-beta' provisioning status is succeeded
VERBOSE: 07:04:22 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-gamma' provisioning status is succeeded
VERBOSE: 07:04:22 PM - Checking deployment status in 5 seconds
VERBOSE: 07:04:27 PM - Checking deployment status in 5 seconds
VERBOSE: 07:04:33 PM - Checking deployment status in 5 seconds
VERBOSE: 07:04:38 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204gamma' provisioning status is succeeded
VERBOSE: 07:04:38 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204beta' provisioning status is succeeded
VERBOSE: 07:04:38 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204alpha' provisioning status is succeeded
VERBOSE: 07:04:38 PM - Checking deployment status in 5 seconds


DeploymentName          : elasticsearch-secure-nodes07212017-110346
ResourceGroupName       : esnodesfbdac204
ProvisioningState       : Succeeded
Timestamp               : 7/23/2017 12:04:32 AM
Mode                    : Incremental
TemplateLink            : 
Parameters              : 
                          Name             Type                       Value     
                          ===============  =========================  ==========
                          admin-username   String                     dbetz     
                          ssh-public-key   String                     ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxbo0LWWXCHEEGxgtraIhHBPPnt+kJGMjYMC6+9gBIsYz8R8bSFfge7ljHRxvJoye+4IrdSf2Ee2grgm2+xT9HjMvVR2/LQjPY+ocdinYHlM6miqvMgMblOMVm6/WwY0L
                          ZkozPKuSXzhO+/Q6HTZBr2pig/bclvJuFPBtClrzZx5R3NfV33/2rZpFZH9OdAf28q55jbZ1t9AJhtD27s34/cRVBXNBQtc2Nw9D8cEJ+raRdJitAOX3U41bjbrO1u3CQ/JtXg/35wZTJH1Yx7zmDl97cklfiArAfaxkgpWkGhob6A6Fu7LvEgLC25gO5NsY+g4CDqGJT5kzbcyQDDh
                          bf dbetz@localhost.localdomain
                          script-base      String                               
                          
Outputs                 : 
DeploymentDebugLogLevel : 

Now we have all kinds of goodies setup. Note how fast that was: template validation was at 07:03:49 PM and it finished at 07:04:38 PM.

post phase 1

Phase 2 Deployment

Now for phase 2. Here we're creating the VPN gateways. Why? Because we have multiple virtual networks in multiple regions. We need to create VPN connections between them to allow communication. To create VPN connections, we need VPN gateways.

Output

Be warned: this takes forever.

(filtering for phase 2)

Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\networkInterfaces\nic-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\networkSecurityGroups\nsg-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\publicIPAddresses\pip-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\virtualNetworks\vnet-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\storage\storageAccounts\storage-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\publicIPAddresses\2.pip-gateway-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\virtualNetworkGateways\2.gateway-copyIndex.json...
(excluding \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\compute\virtualMachines\3.vm-copyIndex.json)
(excluding \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\compute\virtualMachines\extensions\3.script.json)
------------------------------------

------------------------------------
Creating \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\deploy\07222017-074129.2...
Deploying template \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\deploy\azuredeploy-generated.json
VERBOSE: Performing the operation "Creating Deployment" on target "esnodesfbdac204".
VERBOSE: 7:41:42 PM - Template is valid.
VERBOSE: 7:41:43 PM - Create template deployment 'elasticsearch-secure-nodes07222017-074129'
VERBOSE: 7:41:43 PM - Checking deployment status in 5 seconds
VERBOSE: 7:41:49 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-beta' provisioning status is succeeded
VERBOSE: 7:41:49 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-alpha' provisioning status is succeeded
VERBOSE: 7:41:49 PM - Resource Microsoft.Network/virtualNetworks 'vnet-gamma' provisioning status is succeeded
VERBOSE: 7:41:49 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204alpha' provisioning status is succeeded
VERBOSE: 7:41:49 PM - Checking deployment status in 5 seconds
VERBOSE: 7:41:54 PM - Resource Microsoft.Network/networkInterfaces 'nic-gamma' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Network/networkInterfaces 'nic-alpha' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Network/publicIPAddresses 'pip-alpha' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204gamma' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204beta' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Network/publicIPAddresses 'pip-beta' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gamma' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Network/virtualNetworks 'vnet-beta' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Network/virtualNetworks 'vnet-alpha' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-gamma' provisioning status is succeeded
VERBOSE: 7:41:54 PM - Checking deployment status in 5 seconds
VERBOSE: 7:41:59 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-gamma' provisioning status is running
VERBOSE: 7:41:59 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gateway-alpha' provisioning status is succeeded
VERBOSE: 7:41:59 PM - Resource Microsoft.Network/networkInterfaces 'nic-beta' provisioning status is succeeded
VERBOSE: 7:41:59 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gateway-gamma' provisioning status is succeeded
VERBOSE: 7:41:59 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gateway-beta' provisioning status is succeeded
VERBOSE: 7:41:59 PM - Checking deployment status in 10 seconds
VERBOSE: 7:42:09 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-beta' provisioning status is running
VERBOSE: 7:42:09 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-alpha' provisioning status is running
VERBOSE: 7:42:10 PM - Checking deployment status in 11 seconds

takes forever

VERBOSE: 8:11:11 PM - Checking deployment status in 11 seconds
VERBOSE: 8:11:22 PM - Checking deployment status in 11 seconds
VERBOSE: 8:11:33 PM - Checking deployment status in 10 seconds
VERBOSE: 8:11:43 PM - Checking deployment status in 5 seconds
VERBOSE: 8:11:48 PM - Checking deployment status in 9 seconds
VERBOSE: 8:11:58 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-beta' provisioning status is succeeded
VERBOSE: 8:11:58 PM - Checking deployment status in 11 seconds
VERBOSE: 8:12:09 PM - Checking deployment status in 5 seconds
VERBOSE: 8:12:14 PM - Checking deployment status in 7 seconds
VERBOSE: 8:12:21 PM - Checking deployment status in 11 seconds
VERBOSE: 8:12:33 PM - Checking deployment status in 11 seconds
VERBOSE: 8:12:44 PM - Checking deployment status in 5 seconds
VERBOSE: 8:12:49 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-alpha' provisioning status is succeeded
VERBOSE: 8:12:49 PM - Checking deployment status in 8 seconds
VERBOSE: 8:12:57 PM - Checking deployment status in 11 seconds
VERBOSE: 8:13:08 PM - Checking deployment status in 5 seconds
VERBOSE: 8:13:14 PM - Checking deployment status in 7 seconds
VERBOSE: 8:13:21 PM - Checking deployment status in 5 seconds
VERBOSE: 8:13:26 PM - Checking deployment status in 7 seconds
VERBOSE: 8:13:33 PM - Checking deployment status in 11 seconds
VERBOSE: 8:13:45 PM - Checking deployment status in 5 seconds
VERBOSE: 8:13:50 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-gamma' provisioning status is succeeded


DeploymentName          : elasticsearch-secure-nodes07222017-074129
ResourceGroupName       : esnodesfbdac204
ProvisioningState       : Succeeded
Timestamp               : 7/23/2017 1:13:44 AM
Mode                    : Incremental
TemplateLink            : 
Parameters              : 
                          Name             Type                       Value     
                          ===============  =========================  ==========
                          admin-username   String                     dbetz     
                          ssh-public-key   String                     ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxbo0LWWXCHEEGxgtraIhHBPPnt+kJGMjYMC6+9gBIsYz8R8bSFfge7ljHRxvJoye+4IrdSf2Ee2grgm2+xT9HjMvVR2/LQjPY+ocdinYHlM6miqvMgMblOMVm6/WwY0LZkozPKuSXzhO
                          +/Q6HTZBr2pig/bclvJuFPBtClrzZx5R3NfV33/2rZpFZH9OdAf28q55jbZ1t9AJhtD27s34/cRVBXNBQtc2Nw9D8cEJ+raRdJitAOX3U41bjbrO1u3CQ/JtXg/35wZTJH1Yx7zmDl97cklfiArAfaxkgpWkGhob6A6Fu7LvEgLC25gO5NsY+g4CDqGJT5kzbcyQDDhbf 
                          dbetz@localhost.localdomain
                          script-base      String                               
                          
Outputs                 : 
DeploymentDebugLogLevel : 

At this point the gateways have been created.

gws

Creating a VPN Connection Mesh

You can follow whip out various topologies to connect the networks (i.e. hub-and-spoke, point-to-point, etc) . In this case I'm going for a full mesh topology. This connects everyone directly to every one else. It's the most hardcore option.

Given that a connection is unidirectional, you each toplogical unit between areas require both a to and from directional connection. So, A->B and B->A for a 2-point mesh. For a 3-point mesh, it's all over the board. The formula everyone who goes through network engineering training memorizes is n*(n-1). So, for n=3, you have 3 * 2 (6) connections. For n=5, this is 20 connections. That's a lot, but there's no lame bottleneck from tunneling traffic through a central hub (=hub-and-spoke topology).

When creating Azure VPN connections, you specify a shared key. This just makes sense. There needs to be private passcode to enable them to trust each other. In this example, I'm cracking open ASP.NET to auto-generate a wildly complex password. This thing is crazy. Here are some of the passwords it spits out:

  • rT64%nr*#OX/CR)O
  • XwX3UamErI@D)>N{
  • Ej.ZHSngc|yenaiD
  • @*KUz|$#^Jvp-9Vb
  • _7q)6h6/.G;8C?U(

Goodness.

Anyway, on to the races...


    function createmesh { param([Parameter(Mandatory=$true)]$rg,
                                [Parameter(Mandatory=$true)]$key)
    
        function getname { param($id)
            $parts = $id.split('-')
            return $parts[$parts.length-1]
        }
    
        $gateways = Get-AzureRmVirtualNetworkGateway -ResourceGroupName $rg
    
        ($gateways).foreach({
            $source = $_
            ($gateways).foreach({
                $target = $_
                $sourceName = getname $source.Name
                $targetName = getname $target.Name
                if($source.name -ne $target.name) {
                    $connectionName = ('conn-{0}2{1}' -f $sourceName, $targetName)
                    Write-Host "$sourceName => $targetName"
                    New-AzureRmVirtualNetworkGatewayConnection -ResourceGroupName $rg -Location $source.Location -Name $connectionName `
                        -VirtualNetworkGateway1 $source `
                        -VirtualNetworkGateway2 $target `
                        -ConnectionType Vnet2Vnet `
                        -RoutingWeight 10 `
                        -SharedKey $key
                }
            })  
        })
    }
    function _virtualenv {
    
    Add-Type -AssemblyName System.Web
    $key = [System.Web.Security.Membership]::GeneratePassword(16,2)
    
    createmesh -rg $rgGlobal -key $key
    
    } _virtualenv

Output

beta => gamma
beta => alpha
gamma => beta
gamma => alpha
alpha => beta
alpha => gamma



Name                    : conn-beta2gamma
ResourceGroupName       : esnodesfbdac204
Location                : westus
Id                      : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/connections/conn-beta2gamma
Etag                    : W/"d9644745-03f4-461c-9efa-1477cd5e13d1"
ResourceGuid            : bfd89895-af3d-44ee-80c8-74413a18f6c4
ProvisioningState       : Succeeded
Tags                    : 
AuthorizationKey        : 
VirtualNetworkGateway1  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-beta"
VirtualNetworkGateway2  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-gamma"
LocalNetworkGateway2    : 
Peer                    : 
RoutingWeight           : 10
SharedKey               : oF[sa4n^sq)aIYSj
ConnectionStatus        : Unknown
EgressBytesTransferred  : 0
IngressBytesTransferred : 0
TunnelConnectionStatus  : []

Name                    : conn-beta2alpha
ResourceGroupName       : esnodesfbdac204
Location                : westus
Id                      : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/connections/conn-beta2alpha
Etag                    : W/"893dc827-a077-4003-b086-ec3aba8344ee"
ResourceGuid            : c8780c08-9678-4720-a90b-42a8509c059e
ProvisioningState       : Succeeded
Tags                    : 
AuthorizationKey        : 
VirtualNetworkGateway1  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-beta"
VirtualNetworkGateway2  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-alpha"
LocalNetworkGateway2    : 
Peer                    : 
RoutingWeight           : 10
SharedKey               : oF[sa4n^sq)aIYSj
ConnectionStatus        : Unknown
EgressBytesTransferred  : 0
IngressBytesTransferred : 0
TunnelConnectionStatus  : []

Name                    : conn-gamma2beta
ResourceGroupName       : esnodesfbdac204
Location                : eastus
Id                      : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/connections/conn-gamma2beta
Etag                    : W/"a99e47f4-fd53-4583-811f-a868d1c0f011"
ResourceGuid            : 50b8bc36-37b9-434f-badc-961266b19436
ProvisioningState       : Succeeded
Tags                    : 
AuthorizationKey        : 
VirtualNetworkGateway1  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-gamma"
VirtualNetworkGateway2  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-beta"
LocalNetworkGateway2    : 
Peer                    : 
RoutingWeight           : 10
SharedKey               : oF[sa4n^sq)aIYSj
ConnectionStatus        : Unknown
EgressBytesTransferred  : 0
IngressBytesTransferred : 0
TunnelConnectionStatus  : []

Name                    : conn-gamma2alpha
ResourceGroupName       : esnodesfbdac204
Location                : eastus
Id                      : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/connections/conn-gamma2alpha
Etag                    : W/"4dd2d765-4bb0-488f-9d28-dabbf618c28f"
ResourceGuid            : e9e4591f-998b-4318-b297-b2078409c7e9
ProvisioningState       : Succeeded
Tags                    : 
AuthorizationKey        : 
VirtualNetworkGateway1  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-gamma"
VirtualNetworkGateway2  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-alpha"
LocalNetworkGateway2    : 
Peer                    : 
RoutingWeight           : 10
SharedKey               : oF[sa4n^sq)aIYSj
ConnectionStatus        : Unknown
EgressBytesTransferred  : 0
IngressBytesTransferred : 0
TunnelConnectionStatus  : []

Name                    : conn-alpha2beta
ResourceGroupName       : esnodesfbdac204
Location                : centralus
Id                      : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/connections/conn-alpha2beta
Etag                    : W/"aafef4bf-d241-4cdd-88b7-b6ecd793a662"
ResourceGuid            : ef5bb61b-fcbe-4452-bf1f-b847f32dfa95
ProvisioningState       : Succeeded
Tags                    : 
AuthorizationKey        : 
VirtualNetworkGateway1  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-alpha"
VirtualNetworkGateway2  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-beta"
LocalNetworkGateway2    : 
Peer                    : 
RoutingWeight           : 10
SharedKey               : oF[sa4n^sq)aIYSj
ConnectionStatus        : Unknown
EgressBytesTransferred  : 0
IngressBytesTransferred : 0
TunnelConnectionStatus  : []

Name                    : conn-alpha2gamma
ResourceGroupName       : esnodesfbdac204
Location                : centralus
Id                      : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/connections/conn-alpha2gamma
Etag                    : W/"edf5f85a-0f7d-4883-8e45-433de9e045b2"
ResourceGuid            : 074c168c-1d42-4704-b978-124c8505a35b
ProvisioningState       : Succeeded
Tags                    : 
AuthorizationKey        : 
VirtualNetworkGateway1  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-alpha"
VirtualNetworkGateway2  : "/subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodesfbdac204/providers/Microsoft.Network/virtualNetworkGateways/gateway-gamma"
LocalNetworkGateway2    : 
Peer                    : 
RoutingWeight           : 10
SharedKey               : oF[sa4n^sq)aIYSj
ConnectionStatus        : Unknown
EgressBytesTransferred  : 0
IngressBytesTransferred : 0
TunnelConnectionStatus  : []

Phase 3 Deployment

Now to create the VMs...

In our scenario, it's really important to do this phase after creaitng the VPN connection mesh. During the VMs creation, Elasticsearch is automatically setup and the nodes will attempt to connect each other.

No mesh => no connection => you-having-a-fit.

During this deploy, you're going to see that everything from phases 1 and 2 is validated. That's just the idempotent nature of ARM template deployment.

Output

(filtering for phase 3)
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\networkInterfaces\nic-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\networkSecurityGroups\nsg-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\publicIPAddresses\pip-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\virtualNetworks\vnet-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\storage\storageAccounts\storage-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\publicIPAddresses\2.pip-gateway-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\network\virtualNetworkGateways\2.gateway-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\compute\virtualMachines\3.vm-copyIndex.json...
Merging \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\template\resources\compute\virtualMachines\extensions\3.script.json...
Uploading elasticsearch-secure-nodes\07222017-084857\create_data_generation_setup.sh
Uploading elasticsearch-secure-nodes\07222017-084857\install.sh
Uploading elasticsearch-secure-nodes\07222017-084857\generate\hamlet.py
Blob path: https://filesfbdac204.blob.core.windows.net/support/elasticsearch-secure-nodes/07222017-084857
------------------------------------

------------------------------------
Creating \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\deploy\07222017-084857.3...
Deploying template \\10.1.20.1\dbetz\azure\components\elasticsearch-secure-nodes\deploy\azuredeploy-generated.json
VERBOSE: Performing the operation "Creating Deployment" on target "esnodesfbdac204".
VERBOSE: 8:49:02 PM - Template is valid.
VERBOSE: 8:49:03 PM - Create template deployment 'elasticsearch-secure-nodes07222017-084857'
VERBOSE: 8:49:03 PM - Checking deployment status in 5 seconds
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gateway-alpha' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gamma' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-alpha' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gateway-gamma' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/virtualNetworks 'vnet-alpha' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/virtualNetworks 'vnet-beta' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204gamma' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/publicIPAddresses 'pip-gateway-beta' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204alpha' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/publicIPAddresses 'pip-alpha' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Resource Microsoft.Network/publicIPAddresses 'pip-beta' provisioning status is succeeded
VERBOSE: 8:49:08 PM - Checking deployment status in 5 seconds
VERBOSE: 8:49:13 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204beta' provisioning status is succeeded
VERBOSE: 8:49:13 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-beta' provisioning status is running
VERBOSE: 8:49:13 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-beta' provisioning status is succeeded
VERBOSE: 8:49:13 PM - Resource Microsoft.Network/virtualNetworks 'vnet-gamma' provisioning status is succeeded
VERBOSE: 8:49:13 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-alpha' provisioning status is running
VERBOSE: 8:49:13 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204beta' provisioning status is succeeded
VERBOSE: 8:49:13 PM - Resource Microsoft.Network/networkSecurityGroups 'nsg-gamma' provisioning status is succeeded
VERBOSE: 8:49:13 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204gamma' provisioning status is succeeded
VERBOSE: 8:49:13 PM - Resource Microsoft.Network/networkInterfaces 'nic-alpha' provisioning status is succeeded
VERBOSE: 8:49:13 PM - Resource Microsoft.Storage/storageAccounts 'esnodesfbdac204alpha' provisioning status is succeeded
VERBOSE: 8:49:13 PM - Checking deployment status in 6 seconds
VERBOSE: 8:49:19 PM - Resource Microsoft.Compute/virtualMachines 'vm-beta' provisioning status is running
VERBOSE: 8:49:19 PM - Resource Microsoft.Network/networkInterfaces 'nic-gamma' provisioning status is succeeded
VERBOSE: 8:49:19 PM - Resource Microsoft.Compute/virtualMachines 'vm-alpha' provisioning status is running
VERBOSE: 8:49:19 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-gamma' provisioning status is running
VERBOSE: 8:49:19 PM - Resource Microsoft.Network/networkInterfaces 'nic-beta' provisioning status is succeeded
VERBOSE: 8:49:19 PM - Checking deployment status in 11 seconds
VERBOSE: 8:49:30 PM - Resource Microsoft.Compute/virtualMachines 'vm-gamma' provisioning status is running
VERBOSE: 8:49:30 PM - Checking deployment status in 11 seconds
VERBOSE: 8:49:42 PM - Checking deployment status in 11 seconds
VERBOSE: 8:49:53 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-beta' provisioning status is succeeded
VERBOSE: 8:49:53 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-alpha' provisioning status is succeeded
VERBOSE: 8:49:53 PM - Checking deployment status in 11 seconds
VERBOSE: 8:50:04 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:09 PM - Resource Microsoft.Network/virtualNetworkGateways 'gateway-gamma' provisioning status is succeeded
VERBOSE: 8:50:09 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:14 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:19 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:25 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:30 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:35 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:40 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:45 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:51 PM - Checking deployment status in 5 seconds
VERBOSE: 8:50:56 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:01 PM - Resource Microsoft.Compute/virtualMachines/extensions 'vm-gamma/script' provisioning status is running
VERBOSE: 8:51:01 PM - Resource Microsoft.Compute/virtualMachines 'vm-gamma' provisioning status is succeeded
VERBOSE: 8:51:01 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:06 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:11 PM - Resource Microsoft.Compute/virtualMachines 'vm-alpha' provisioning status is succeeded
VERBOSE: 8:51:11 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:16 PM - Resource Microsoft.Compute/virtualMachines/extensions 'vm-beta/script' provisioning status is running
VERBOSE: 8:51:16 PM - Resource Microsoft.Compute/virtualMachines/extensions 'vm-alpha/script' provisioning status is running
VERBOSE: 8:51:16 PM - Resource Microsoft.Compute/virtualMachines 'vm-beta' provisioning status is succeeded
VERBOSE: 8:51:16 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:22 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:27 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:32 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:37 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:43 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:48 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:53 PM - Checking deployment status in 5 seconds
VERBOSE: 8:51:58 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:03 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:08 PM - Resource Microsoft.Compute/virtualMachines/extensions 'vm-gamma/script' provisioning status is succeeded
VERBOSE: 8:52:08 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:14 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:19 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:24 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:29 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:35 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:40 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:45 PM - Resource Microsoft.Compute/virtualMachines/extensions 'vm-beta/script' provisioning status is succeeded
VERBOSE: 8:52:45 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:50 PM - Checking deployment status in 5 seconds
VERBOSE: 8:52:55 PM - Resource Microsoft.Compute/virtualMachines/extensions 'vm-alpha/script' provisioning status is succeeded


DeploymentName          : elasticsearch-secure-nodes07222017-084857
ResourceGroupName       : esnodesfbdac204
ProvisioningState       : Succeeded
Timestamp               : 7/23/2017 1:52:51 AM
Mode                    : Incremental
TemplateLink            : 
Parameters              : 
                          Name             Type                       Value     
                          ===============  =========================  ==========
                          admin-username   String                     dbetz     
                          ssh-public-key   String                     ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxbo0LWWXCHEEGxgtraIhHBPPnt+kJGMjYMC6+9gBIsYz8R8bSFfge7ljHRxvJoye+4IrdSf2Ee2grgm2+xT9HjMvVR2/LQjPY+ocdinYHlM6miqvMgMblOMVm6/WwY0LZkozPKuSXzhO
                          +/Q6HTZBr2pig/bclvJuFPBtClrzZx5R3NfV33/2rZpFZH9OdAf28q55jbZ1t9AJhtD27s34/cRVBXNBQtc2Nw9D8cEJ+raRdJitAOX3U41bjbrO1u3CQ/JtXg/35wZTJH1Yx7zmDl97cklfiArAfaxkgpWkGhob6A6Fu7LvEgLC25gO5NsY+g4CDqGJT5kzbcyQDDhbf 
                          dbetz@localhost.localdomain
                          script-base      String                     https://filesfbdac204.blob.core.windows.net/support/elasticsearch-secure-nodes/07222017-084857
                          
Outputs                 : 
DeploymentDebugLogLevel : 

~4 minutes total to setup 3 VMs is pretty good. Keep in mind that this time frame includes running the post-VM-creation script (install.sh). I loaded that with a bunch of stuff. You can see this part of the deploy in lines like the following:

Resource Microsoft.Compute/virtualMachines/extensions 'vm-beta/script' provisioning status is running

and

Resource Microsoft.Compute/virtualMachines/extensions 'vm-gamma/script' provisioning status is succeeded

Why's it so fast? Two reasons: First, the storage accounts, VNets, IPs, NICs, and NSGs are already setup in phase 1. Second, Azure will parallel deploy whatever it can. Upon validating that the dependencies (dependsOn) are already setup, Azure will deploy the VMs. This means that phase 3 is a parallel deployment of three VMs.

Inspection

At this point the entire core infrastructure is in place, including VMs. We can verify this by looking at the Elasticsearch endpoint.

While we can easily derive the endpoint address, let's let Powershell tell us directly:


    (Get-AzureRmPublicIpAddress -ResourceGroupName $rgGlobal -Name "pip-alpha").DnsSettings.Fqdn 
    (Get-AzureRmPublicIpAddress -ResourceGroupName $rgGlobal -Name "pip-beta").DnsSettings.Fqdn 
    (Get-AzureRmPublicIpAddress -ResourceGroupName $rgGlobal -Name "pip-gamma").DnsSettings.Fqdn 

From this we have the following:

esnodesfbdac204-alpha.centralus.cloudapp.azure.com
esnodesfbdac204-beta.westus.cloudapp.azure.com
esnodesfbdac204-gamma.eastus.cloudapp.azure.com

With this we can access the following endpoints to see pairing. You only need to do this on one, but because these are public, let's look at all three:

http://esnodesfbdac204-alpha.centralus.cloudapp.azure.com:9200
http://esnodesfbdac204-beta.westus.cloudapp.azure.com:9200
http://esnodesfbdac204-gamma.eastus.cloudapp.azure.com:9200

Always use SSL. Consider HTTP deprecated.

nodes

Let's see if we have any indices.

w/o data

Nope. That makes sense because... well... I didn't create any yet.

Logging into alpha

We want to generate some data for Elasticsearch. I've provided a generation tool which the VMs setup during their provisioning.

Before we get to that point, we have to login to a VM.

Choose a VM DNS name and try to ssh to it. I don't care which one. I'm going with alpha.

[dbetz@core ~]$ ssh esnodesfbdac204-alpha.westus.cloudapp.azure.com
The authenticity of host 'esnodesfbdac204-alpha.centralus.cloudapp.azure.com (52.165.135.82)' can't be established.
ECDSA key fingerprint is 36:d7:fd:ab:39:b1:10:c2:88:9f:7a:87:30:15:8f:e6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'esnodesfbdac204-alpha.centralus.cloudapp.azure.com,52.165.135.82' (ECDSA) to the list of known hosts.
[dbetz@alpha ~]$

Troubleshooting

If you get a message like the following, then you don't have the private key that goes with the public key you gave the VM in the ARM template.

[dbetz@core ~]$ ssh esnodesfbdac204-alpha.westus.cloudapp.azure.com
The authenticity of host 'esnodesfbdac204-alpha.westus.cloudapp.azure.com (13.87.182.255)' can't be established.
ECDSA key fingerprint is 94:dd:1b:ca:bf:7a:fd:99:c2:70:02:f3:0c:fa:0b:9a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'esnodesfbdac204-alpha.westus.cloudapp.azure.com,13.87.182.255' (ECDSA) to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

To get your private key you can dump it out:

[dbetz@core ~]$ cat ~/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
...base64 stuff here...
-----END RSA PRIVATE KEY-----

You can take that and dump it into a different system:

[dbetz@core ~]$ export HISTCONTROL=ignorespace
[dbetz@core ~]$         cat > ~/.ssh/id_rsa <<\EOF
-----BEGIN RSA PRIVATE KEY-----
...base64 stuff here...
-----END RSA PRIVATE KEY-----
EOF

When HISTCONTROL is set to ignorespace is set and a command has a space in front of it, it won't be stored in shell history.

When you try it again, you'll get a sudden urge to throw your chair across the room:

[dbetz@core ~]$ ssh esnodesfbdac204-alpha.eastus.cloudapp.azure.com       
The authenticity of host 'esnodesfbdac204-alpha.eastus.cloudapp.azure.com (13.87.182.255)' can't be established.
ECDSA key fingerprint is 94:dd:1b:ca:bf:7a:fd:99:c2:70:02:f3:0c:fa:0b:9a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'esnodesfbdac204-alpha.eastus.cloudapp.azure.com,13.87.182.255' (ECDSA) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0664 for '/home/dbetz/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /home/dbetz/.ssh/id_rsa
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Chill. Your permissions suck. File permissions are set via your umask settings. In this case, they're too open. Only you need permissions, read-only permissions at that: 400.

You just need to drop the permissions:

[dbetz@core ~]$ chmod 400 ~/.ssh/id_rsa 

Now you can get in:

[dbetz@core ~]$ ssh esnodesfbdac204-alpha.eastus.cloudapp.azure.com  
The authenticity of host 'linux04.jampad.net (192.80.189.178)' can't be established.
ECDSA key fingerprint is 7a:24:38:8c:05:c1:2c:f3:d0:fa:52:0d:2c:a4:04:9c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'linux04.jampad.net,192.80.189.178' (ECDSA) to the list of known hosts.
Last login: Sat Jul 22 17:39:07 2017 from 136.61.130.214
[dbetz@alpha ~]$ 

We're in.

Adding data

To generate sample data for this scenario run my data generation tool (based on my Github project Hamlet.

It's in the /home/root folder.

[root@alpha ~]# ./setup_data_generation.sh

Azure does its automated setup as root. That's why were here.

Running this installs all the required tool and writes instructions.

Follow the instructions the tool provides :

[root@alpha ~]# cd /srv/hamlet
[root@alpha hamlet]# source bin/activate
(hamlet) [root@alpha hamlet]# cd content
(hamlet) [root@alpha content]# python /srv/hamlet/content/hamlet.py
http://10.17.0.4:9200/librarygen
^CStopped (5.8595204026280285)

I let it run for a few seconds, then hit CTRL-C to exit.

Now let's refresh our Elasticsearch endpoints (the :9200 endpoints).

with data

The data is there and has replicated across all servers.

Looking at all three systems at once is just for the purpose of this demo. In reality, all you have to do is look at /_cat/shards on any node ins the cluster and be done with it:

shards

You can even see that there are multiple shards and replicas (p => primary, r => replica).

Create Traffic Manager

At this point we want to create a single point-of-contact for search. We do this with traffic manager. You create the traffic manager then add an endpoint for each system:

    function createtrafficmanager { param([Parameter(Mandatory=$true)]$rg,
                                          [Parameter(Mandatory=$true)]$count)
        clear
            
        $names = @("alpha", "beta", "gamma", "delta", "epsilon")
    
        $uniqueName = (Get-AzureRmStorageAccount -ResourceGroupName $rg)[0].StorageAccountName
    
        $tmProfile = New-AzureRmTrafficManagerProfile -ResourceGroupName $rg -name "tm-$rg" `
                        -TrafficRoutingMethod Performance `
                        -ProfileStatus Enabled `
                        -RelativeDnsName $uniqueName `
                        -Ttl 30 `
                        -MonitorProtocol HTTP `
                        -MonitorPort 9200 `
                        -MonitorPath "/"
    
        (1..$count).foreach({
            $name = $names[$_ - 1]
            $pip = Get-AzureRmPublicIpAddress -ResourceGroupName $rg -Name "pip-$name"
            Add-AzureRmTrafficManagerEndpointConfig -TrafficManagerProfile $tmProfile -EndpointName $name -TargetResourceId $pip.id -Type AzureEndpoints -EndpointStatus Enabled
        })
        Set-AzureRmTrafficManagerProfile -TrafficManagerProfile $tmProfile
        
    }
    
    createtrafficmanager -rg 'esnodes4dede7b0' -count 3

Output

Id                               : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodes4dede7b0/providers/Microsoft.Network/trafficManagerProfiles/tm-esnodes4dede7b0
Name                             : tm-esnodes4dede7b0
ResourceGroupName                : esnodes4dede7b0
RelativeDnsName                  : esnodes4dede7b0alpha
Ttl                              : 30
ProfileStatus                    : Enabled
TrafficRoutingMethod             : Performance
MonitorProtocol                  : HTTP
MonitorPort                      : 9200
MonitorPath                      : /
MonitorIntervalInSeconds         : 30
MonitorTimeoutInSeconds          : 10
MonitorToleratedNumberOfFailures : 3
Endpoints                        : {alpha, beta, gamma}

Id                               : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodes4dede7b0/providers/Microsoft.Network/trafficManagerProfiles/tm-esnodes4dede7b0
Name                             : tm-esnodes4dede7b0
ResourceGroupName                : esnodes4dede7b0
RelativeDnsName                  : esnodes4dede7b0alpha
Ttl                              : 30
ProfileStatus                    : Enabled
TrafficRoutingMethod             : Performance
MonitorProtocol                  : HTTP
MonitorPort                      : 9200
MonitorPath                      : /
MonitorIntervalInSeconds         : 30
MonitorTimeoutInSeconds          : 10
MonitorToleratedNumberOfFailures : 3
Endpoints                        : {alpha, beta, gamma}

Id                               : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodes4dede7b0/providers/Microsoft.Network/trafficManagerProfiles/tm-esnodes4dede7b0
Name                             : tm-esnodes4dede7b0
ResourceGroupName                : esnodes4dede7b0
RelativeDnsName                  : esnodes4dede7b0alpha
Ttl                              : 30
ProfileStatus                    : Enabled
TrafficRoutingMethod             : Performance
MonitorProtocol                  : HTTP
MonitorPort                      : 9200
MonitorPath                      : /
MonitorIntervalInSeconds         : 30
MonitorTimeoutInSeconds          : 10
MonitorToleratedNumberOfFailures : 3
Endpoints                        : {alpha, beta, gamma}

Id                               : /subscriptions/20e08d3d-d5c5-4f76-a454-4a1b216166c6/resourceGroups/esnodes4dede7b0/providers/Microsoft.Network/trafficManagerProfiles/tm-esnodes4dede7b0
Name                             : tm-esnodes4dede7b0
ResourceGroupName                : esnodes4dede7b0
RelativeDnsName                  : esnodes4dede7b0alpha
Ttl                              : 30
ProfileStatus                    : Enabled
TrafficRoutingMethod             : Performance
MonitorProtocol                  : HTTP
MonitorPort                      : 9200
MonitorPath                      : /
MonitorIntervalInSeconds         : 30
MonitorTimeoutInSeconds          : 10
MonitorToleratedNumberOfFailures : 3
Endpoints                        : {alpha, beta, gamma}

Just access your endpoint:

http://esnodes4dede7b0.trafficmanager.net:9200

shards

Now you have a central endpoint. In this case traffic will be send to whichever Elasticsearch endpooint is closest to the end user (=performance traffic routing method).

You'll want SSL for that. Well, after you go through the following DNS section.

Adding DNS

At this point everything is functional. Let's go beyond bare functionality. Normally, you'd want something like search.davidbetz.net as an endpoint.

The following will give my davidbetz.net domain a subdomain: esnodes4dede7b0.davidbetz.net.


    function creatednscname { param([Parameter(Mandatory=$true)]$dnsrg,
                                          [Parameter(Mandatory=$true)]$zonename,
                                          [Parameter(Mandatory=$true)]$cname,
                                          [Parameter(Mandatory=$true)]$target)
    
        New-AzureRmDnsRecordSet -ResourceGroupName $dnsrg -ZoneName $zonename -RecordType CNAME -Name $cname -Ttl 3600 -DnsRecords (
            New-AzureRmDnsRecordConfig -Cname $target
        )
    }
    
    function _virtualenv {
    
    $dnsrg = 'davidbetz01'
    $zone = 'davidbetz.net'
    
    creatednscname $dnsrg $zone $rgGlobal "$rgGlobal.trafficmanager.net"
    
    } _virtualenv

Done.

shards

Addition Thoughts

Remember, SSL. You do this with SSL termination. Putting SSL between each system internally (between nginx and an internal web server) is naive and foolish; you only need SSL to external systems. You do this with Nginx. See my secure Elasticsearch lab at https://linux.azure.david.betz.space/_/elasticsearch-secure for details.

You'll also want to protect various Elasticsearch operations via password (or whatever). See my Running with Nginx for more information.

You can learn more about interacting with Elasticsearch directly via my article Learning Elasticsearch with PowerShell.

1 2

Powered by
Python / Django / Elasticsearch / Azure / Nginx / CentOS 7

Mini-icons are part of the Silk Icons set of icons at famfamfam.com