2005 2006 2007 2008 2009 2010 2011 2015 2016 2017 aspnet azure csharp debugging elasticsearch exceptions firefox javascriptajax linux llblgen mongodb powershell projects python security services silverlight training videos wcf wpf xag xhtmlcss

It's a NoSQL database

10 minutes after I did a webinar, explaining how we need to be specific about database storage and never use the unbelievably unhelpful and pointless term "NoSQL".

What I asked: What can you tell me about that type of database?

Response: It's a NoSQL database.



What I meant: What kind of structure does this database use? Relational? Hierarchical? Document? Graph? Key-value? Column-storage? Wide table? What kind of internal storage does this database use? B-tree? BW-tree? Memory mapping? Hash tables with linked lists? Is the data per collection/table or is that an abstraction for underlying partitions? Does the data format lend itself to easy horizontal partitioning (e.g. sharding)? If it's a sharded system, how does it deal with the scatter-gather problem and multi-node write confirmations? Is there a data router? Are replicas readable? Are there capabilities for edge nodes? How is data actually tracked (e.g. bitmaps, statistics)? Are records/documents mutable or immutable with delta structures? Are records deleted immediately or internally marked for deletion? What are this system's memory management strategy? What kind of aggregations are possible with this database? Are there silly 100MB aggregation limits or is it willing to use actually use the 128GB of RAM we just paid for specifically for the purpose of preventing our massive aggregates from needing to write to disk? Is there a transaction / an operations log? Is there support for transactions? How does this relate to rollbacks and commits? Are there various data isolation levels? Is locking pessimistic, optimistic, or just configurable? What can you tell me about available local partitioning strategies? Can you partition indexes to various local disks to optimize performance? What mechanisms allow us to optimize reads and writes separately? Is there native encryption at the database, table, field, or column level? Are there variable storage engines for this database? At what various levels does compression take place?

What I was told: You don't access it with SQL.





confused

PowerShelling Azure DNS Management

DNS is one of those topics that every developer has to deal with. It's also one of those topics some of us developers get to deal with. I love DNS. It allows all manner of flexible addressing, failover, and load-balancing techniques. Understanding DNS takes you a long way in a whole lot of scenarios.

However, the difference between the fun architecture and possibilities of DNS and the implementation has often been the difference between a great sounding class syllabus and the horrible reality of a boatload of homework. Windows DNS, for example, was historically cryptic GUI gibberish. BIND was OK and has a very simple config format, but it scared most people off (you simply cannot win with people who hate GUIs and command lines).

Azure DNS lets you config via the GUI (noooooob; jk, the GUI is pretty nice), ARM templates, Azure CLI (=Linux), and PowerShell.

But, HOW do you manage the config? What format? How do you deploy?

One way is to do everything in ARM templates. Check out this example:


  "resources": [
    {
      "type": "Microsoft.Network/dnszones",
      "name": "example.org",
      "apiVersion": "2016-04-01",
      "location": "global",
      "properties": { }
    },
    {
      "type": "Microsoft.Network/dnszones/a",
      "name": "example.org/mysubdomain01",
      "apiVersion": "2016-04-01",
      "location": "global",
      "dependsOn": [
        "example.org"
      ],
      "properties": {
        "TTL": 3600,
        "ARecords": [{
            "ipv4Address": "127.0.0.1"
          }
        ]
      }
    }
  ],


There's nothing wrong with that. If you want to have a billion lines of JSON for the sake of a single DNS record, you have fun with that. Ugh. Fine. Yes, it will be idempotent, but are severe overkill for quick updates.

Because PowerShell is effectively the .NET REPL, you can write a simple PowerShell (=.NET) tool to handle any custom format you want.

The following is one way of formatting your DNS updates:


    @(
        @{
            Name="example1"
            Config=@(
            @{
                Type="CNAME"
                Value="hosting01gyfir-alpha.centralus.cloudapp.azure.com"
            })
        },
        @{
            Name= "example2"
            Config=@(
            @{
                Type="A"
                Value="127.0.0.1"
            })
        },
        @{
            Name= "mail"
            Config=@(
            @{
                Type="MX"
                Preference=10
                Exchange="mail.example.com"
            },
            @{
                Type="MX"
                Preference=20
                Exchange="mail2.example.org"
            })
        }
    )


I donno. I find that pretty simple. I'll use that when I setup something new.

Here's my function (with call) to make that work:


    function deployDns { param([Parameter(Mandatory=$true)]$rg, [Parameter(Mandatory=$true)]$zonename, [Parameter(Mandatory=$true)] $ttl, [Parameter(Mandatory=$true)]$records, [bool]$testOnly)
        ($records).foreach({
            $name = $_.Name
            Write-Host "Name: $name"
            $dnsrecords = @()
    
            ($_.Config).foreach({
                $config = $_
                $type = $config.Type
                switch($type) {
                    "CNAME" {
                        Write-Host ("`tCNAME: {0}" -f $config.Value)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Cname $config.Value
                    }
                    "MX" {
                        Write-Host ("`tPreference: {0}" -f $config.Preference)
                        Write-Host ("`tExchange: {0}" -f $config.Exchange)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Preference $config.Preference -Exchange $config.Exchange
                    }
                    "A" {
                        Write-Host ("`tPreference: {0}" -f $config)
                        Write-Host ("`tExchange: {0}" -f $config.Ipv4Address)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Ipv4Address $config.Value
                    }
                    "AAAA" {
                        Write-Host ("`tIpv6Address: {0}" -f $config.Ipv6Address)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Ipv6Address $config.Value
                    }
                    "NS" {
                        Write-Host ("`tNS: {0}" -f $config.Value)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Nsdname $config.Value
                    }
                    "PTR" {
                        Write-Host ("`tPtrdname: {0}" -f $config.Value)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Ptrdname $config.Value
                    }
                    "TXT" {
                        Write-Host ("`tPtrdname: {0}" -f $config.Value)
                        $dnsrecords += New-AzureRmDnsRecordConfig -Value $config.Value
                    }
                }
            })
            Write-Host
            Write-Host "Records:"
            Write-Host $dnsrecords
            if(!$testOnly) {
                New-AzureRmDnsRecordSet -ResourceGroupName $rg -ZoneName $zonename -RecordType $type -Name $name -Ttl $ttl -DnsRecords @dnsrecords
            }
            Write-Host
            Write-Host
        })
    }
    
    deployDns -testOnly $true -rg 'davidbetz01' -zonename "davidbetz.net" -ttl 3600 -records @(
        @{
            Name="example1"
            Config=@(
            @{
                Type="CNAME"
                Value="hosting01gyfir-alpha.centralus.cloudapp.azure.com"
            })
        },
        @{
            Name= "example2"
            Config=@(
            @{
                Type="A"
                Value="127.0.0.1"
            })
        },
        @{
            Name= "mail"
            Config=@(
            @{
                Type="MX"
                Preference=10
                Exchange="mail.example.com"
            },
            @{
                Type="MX"
                Preference=20
                Exchange="mail2.example.org"
            })
        }
    )


Is this insane? Probably. That's what I'm known for. ARM templates might be smarter given their idempotent nature, and I've found myself using the GUI now and again.

For now, just keep in mind that PowerShell lets you me ultra flexible with your Azure configuration, not just Azure DNS.

Allowing access to your Azure VM

You created your VM, you installed and configured all services, and your firewalld/iptables is set correctly. Your nmap tests are even working between systems.

But, you can't access your services external to Azure?

You probably didn't enable access in Azure. You need to allow specific ports in your Azure Network Security Group.

In terms of your Azure objects, your VM uses a NIC, your NIC uses an NSG.

Using PowerShell

Using PowerShell, you can do something like this:

    
    $rg = 'hosting01'
    
    $nsg = Get-AzureRmNetworkSecurityGroup -ResourceGroupName $rg -Name "$rg-nsg-alpha"
    
    $maximum = ($nsg.SecurityRules | measure -Property priority -Maximum).Maximum + 100
    $httpRule = New-AzureRmNetworkSecurityRuleConfig -Name "http" -Protocol Tcp -SourceAddressPrefix * -DestinationAddressPrefix * -SourcePortRange * -DestinationPortRange 80 -Priority $maximum -Description "HTTP" -Direction Inbound -Access Allow
    $nsg.SecurityRules.Add($httpRule)
    
    $maximum = ($nsg.SecurityRules | measure -Property priority -Maximum).Maximum + 100
    $httpsRule = New-AzureRmNetworkSecurityRuleConfig -Name "https" -Protocol Tcp -SourceAddressPrefix * -DestinationAddressPrefix * -SourcePortRange * -DestinationPortRange 443 -Priority $maximum -Description "SSL" -Direction Inbound -Access Allow
    $nsg.SecurityRules.Add($httpsRule)
    
    Set-AzureRmNetworkSecurityGroup -NetworkSecurityGroup $nsg


Use an ARM Template

Or you can just fix your initial ARM template by adding the resource:

See the https://linux.azure.david.betz.space/_/python-uwsgi-nginx on https://linux.azure.david.betz.space for a fuller example.


    {
        "comments": "",
        "type": "Microsoft.Network/networkSecurityGroups",
        "name": "nsg-alpha",
        "apiVersion": "2017-03-01",
        "location": "[resourceGroup().location]",
        "properties": {
            "securityRules": [
                {
                    "name": "default-allow-ssh",
                    "properties": {
                        "protocol": "Tcp",
                        "sourcePortRange": "*",
                        "destinationPortRange": "22",
                        "sourceAddressPrefix": "*",
                        "destinationAddressPrefix": "*",
                        "access": "Allow",
                        "priority": 1000,
                        "direction": "Inbound"
                    }
                },
                {
                    "name": "http",
                    "properties": {
                        "protocol": "Tcp",
                        "sourcePortRange": "*",
                        "destinationPortRange": "80",
                        "sourceAddressPrefix": "*",
                        "destinationAddressPrefix": "*",
                        "access": "Allow",
                        "priority": 1100,
                        "direction": "Inbound"
                    }
                },
                {
                    "name": "https",
                    "properties": {
                        "protocol": "Tcp",
                        "sourcePortRange": "*",
                        "destinationPortRange": "443",
                        "sourceAddressPrefix": "*",
                        "destinationAddressPrefix": "*",
                        "access": "Allow",
                        "priority": 1200,
                        "direction": "Inbound"
                    }
                }
            ]
        },
        "resources": [],
        "dependsOn": []
    }


Add this to your NICs (Microsoft.Network/networkInterfaces) properties:

    
    "type": "Microsoft.Network/networkInterfaces",
    "properties": {
        "networkSecurityGroup": {
            "id": "[resourceId('Microsoft.Network/networkSecurityGroups', concat(variables('nsg-prefix'), variables('names')[0]))]"
        }
    }


...and dependsOn section:


    
    "dependsOn": [
        "[resourceId('Microsoft.Network/networkSecurityGroups', concat(variables('nsg-prefix'), variables('names')[0]))]"
    ]


Using Azure DNS

Though you might be used to doing DNS via bind, there's nothing specific about bind that defines DNS. You can do DNS anywhere; all DNS details are blackboxed.

As such, Azure DNS is perfectly compatible with being a super Linux geek.

You can review the docs for Azure DNS on your own, but I'd like to provide a few "recipes".

Everything that follows assumes that you've done the prerequisite work of getting an active Azure account, creating your resource group (New-AzureRmResourceGroup) and creating a new DNS zone (New-AzureRmDnsZone).


New-AzureRmResourceGroup -Name spartan01
New-AzureRmDnsZone -ResourceGroupName spartan01 -Name "example.com"

In moving to Azure DNS, you have a few things that you need to do. These have NOTHING to do with Azure -- they're just facts of DNS.

PowerShell will be used for the most part, but the Azure CLI syntax will be provided where it's particularly interesting.

E-Mail (MX Records)

First, you need to realize that DNS affects your e-mail. This should be obvious since your e-mail has a domain name right in it. So, you need to make sure your MX records are legit.

Here's a block of PowerShell (UpdateAzureDNSMX.ps1) you should keep around:


    New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType A -Name "mail" -Ttl 3600 -DnsRecords (New-AzureRmDnsRecordConfig -Ipv4Address '203.0.113.1')
    
    New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType MX -Name "mail" -Ttl 3600 -DnsRecords @(
        (New-AzureRmDnsRecordConfig -Preference 10 -Exchange smtp.example.com),
        (New-AzureRmDnsRecordConfig -Preference 20 -Exchange mailstore1.example.com)
    )

This creates a record set, then adds the specific config.

Single IP Address

Let's say you have a bunch of sites and want a single server to handle them all. Let's also assume that you only have one IP address.

You should look into using WebApps, but let's pretend you absolutely need a VM for some wild reason.

What to do?

Simple: CNAME it.

After creating your shiny new VM (perhaps running ./create simple hosting01 from https://linux.azure.david.betz.space), you need to get the systems IP address:

Azure CLI

[dbetz@callisto ~]$ az network public-ip list --query "[?dnsSettings.domainNameLabel!=null]|[?starts_with(dnsSettings.domainNameLabel,'hosting01')].{ dns: dnsSettings.fqdn  }"           
[
  {
    "dns": "hosting01figrg-alpha.centralus.cloudapp.azure.com"
  }
]

PowerShell


    (Get-AzureRmPublicIpAddress -ResourceGroupName hosting01 -Name hosting01-ip-alpha).DnsSettings.Fqdn

Let's use that Azure domain name to create our CNAME records for our domains:


    function create { param($name)
        New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType CNAME -Name $name -Ttl 3600 -DnsRecords (New-AzureRmDnsRecordConfig -Cname "hosting01figrg-alpha.centralus.cloudapp.azure.com")
    }
    create "subdomain01" 
    create "subdomain02" 
    create "subdomain03" 
    create "subdomain04" 

For the sake of this example, all domains will be as subdomains. Doing full, separate domains is the same, but the examples would be longer... and less fun. It's the same idea, though.

If you already have the records (e.g. you move to another VM later), you update:

    
    function update { param($name)
        $rs = Get-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com"  -RecordType CNAME -Name $name
        $rs.Records[0] = (New-AzureRmDnsRecordConfig -Cname "hosting01figrg-alpha.centralus.cloudapp.azure.com")
        Set-AzureRmDnsRecordSet -RecordSet $rs
    }
    update "subdomain01" 
    update "subdomain02" 
    update "subdomain03" 
    update "subdomain04" 

Now you have your domains all pointing to the same place.

Stopping here doesn't help you. You still have to figure out how to get your server to handle the multiple sites! Though it's not directly Azure related, I find partial examples to be distasteful and a sign of extreme ignorance on the part of an author.

So, what do you do on the server?

Let's assume you're reading this after 2016 (I'm writing this in 2017, sooooo...), therefore you're not using Apache. You're using Nginx:

server {
    listen 80;

    server_name mysubdomain01.example.com;
    
    return 301 https://mysubdomain01.example.com$request_uri;
}

server {
    listen 443 ssl http2;

    server_name mysubdomain01.example.com;

    ssl_certificate /etc/letsencrypt/live/mysubdomain01.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysubdomain01.example.com/privkey.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

    ssl_prefer_server_ciphers on;

    ssl_dhparam /srv/_cert/dhparam.pem;

    location / {
        add_header Strict-Transport-Security max-age=15552000;
        proxy_pass http://127.0.0.1:8081;        
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Port 443;
        proxy_set_header Host mysubdomain01.example.com;
    }
}

I'm not into trivial examples, so the full SSL example is provided, with the letsencrypt paths.

The point is this:

  • listen on the port without the IP
  • server the server_name to your full domain name

Done.

Well, no, we're not done to my satisfaction. You still need some ability to test this. So, here's a quick Node.js application you can use to listen on port 80. Just curl from somewhere else and see if it works.

curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -
yum -y install nodejs

cd
cat > server.js << EOF
http = require('http');
port = parseInt(process.argv[2]);
server = http.createServer( function(req, res) {
    res.writeHead(200, {'Content-Type': 'text/html'});
    res.end(req.method + ' server ' + port);
});
host = '$PUBLIC_IP';
server.listen(port, host);
EOF

node server.js 8081 &

Azure CDN

When using assets (e.g. png, css) on your website, you'll want to be sure to serve them via an Azure CDN.

I tend to avoid the term "blob" as that's a binary entity -- the moment you start talking about "SVG blobs" or "CSS blobs", you lose any connection to reality. They're assets.

If your CDNS provider (Azure uses both Akamai and Verizon) supports SSL for custom domains, then great. If not, you're stuck with the mycdnendpoint.azurewebsites.net address. You can consider HTTP obsolete. Use SSL or do not host anything.

Assuming your CDN provider supports SSL for custom domains, you have to tell it about your custom domain.

It requires that you prove ownership by creating a cdnverify CNAME entry. Most of the docs are deeply cryptic on this (honestly, RTFM means FU; it's rarely helpful), so I'll make it simple:

if you want to use cdn01.mydomain.net instead of http://mycdnendpoint.azurewebsites.net, you can do this in two ways (choose one):

  • create a cdn01 CNAME pointing to mycdnendpoint.azurewebsites.net

New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType CNAME -Name 'cd01' -Ttl 3600 -DnsRecords (New-AzureRmDnsRecordConfig -Cname "mycdnendpoint.azureedge.net")


OR

  • create a cdnverify.cdn01 CNAME pointing to cdnverify.mycdnendpoint.azurewebsites.net.

New-AzureRmDnsRecordSet -ResourceGroupName spartan01 -ZoneName "example.com" -RecordType CNAME -Name 'cdnverify.cd01' -Ttl 3600 -DnsRecords (New-AzureRmDnsRecordConfig -Cname "cdnverify.mycdnendpoint.azureedge.net")


NS Records

Once your DNS is setup and ready for production, you need to tell your registrar about it.

You can't guess what these records are. You just ask Azure about them:


    (Get-AzureRmDnsZone -ResourceGroupName spartan01 -Name "example.com").NameServers

Here's a preview of something from namecheap.com (use whatever you want... as long as it's not GoDaddy)

ARM Components

Generating MongoDB Sample Data

Last year I wrote about generating better random strings.

Lorem Ipsum is the devil. It messes with us who are students of Latin; Cicero is hard enough without people throwing randomized Cicero in our faces. It's better to use something that isn't part of a linguistic insurgency. Use my Hamlet generator instead.

Anyway...

Because MongoDB is a standard component of any modern architecture these days, we need the ability to generate, not simply strings, but full objects for our test databases.

The following MongoDB script will do just that. Change the value of the run function-call to set the number of objects to throw at MongoDB.

You run this with the MongoDB shell:

./mongo < hamlet.js

Note: The third-party tool Robomongo, while awesome for day-to-day usage, will not work for this. It doens't play nicely with initializeUnorderedBulkOp, which you need for bulk data import. It's like the BULK INSERT command in SQL.

You can use the following with abridged data or this with the full hamlet lexicon.

var raw = "o my offence is rank it smells to heaven hath the primal eldest curse upont a brothers murder pray can i not though inclination be as sharp will stronger guilt defeats strong intent and like man double business bound stand in pause where shall first begin both neglect what if this cursed hand were thicker than itself with blood there rain enough sweet heavens wash white snow whereto serves mercy but confront visage of whats prayer two-fold force forestalled ere we come fall or pardond being down then ill look up fault past form serve turn forgive me foul that cannot since am still possessd those effects for which did crown mine own ambition queen may one retain corrupted currents world offences gilded shove by justice oft tis seen wicked prize buys out law so above no shuffling action lies his true nature ourselves compelld even teeth forehead our faults give evidence rests try repentance can yet when repent wretched state bosom black death limed soul struggling free art more engaged help angels make assay bow stubborn knees heart strings steel soft sinews newborn babe all well";
var data = raw.split(" ");

function hamlet(count) {
    return data[parseInt(Math.random() * data.length)] + (count == 1 ? "" : " " + hamlet(count - 1));
}

function randrange(min, max) {
    if(!max) { max = min; min = 1;}
    return Math.floor(Math.random() * (max - min + 1)) + min;
}

function createArray(count, generator) {
    var list = [];
    for(var n=0; n<count; n++) {
        list.push(generator());
    }
    return list;
}

function pad(number){
    return ("0" + number).substr(-2);
}

function createItem() {
    item = {
        "_id": "9780" + randrange(100000000, 999999999),
        "title": hamlet(randrange(4, 8)),
        "authors": createArray(randrange(4), function() { return hamlet(2) }),
        "metadata": {
            "pages": NumberInt(randrange(1, 400)),
            "genre": createArray(randrange(2), function() { return hamlet(1) }),
            "summary": hamlet(randrange(100, 400)),
        },
        "published": new Date(randrange(1960, 2016) + "-" + pad(randrange(12)) + "-" + pad(randrange(28)))
    };

    if (randrange(4) == 1) {
        item.editor = hamlet(1);
    }

    return item;
}

function run(count) {
    var bulk = db.book.initializeUnorderedBulkOp();
    for (var n = 0; n < count; n++) {
        bulk.insert(createItem());
    }
    bulk.execute();
}

run(100000)

Powered by
Python / Django / Elasticsearch / Azure / Nginx / CentOS 7

Mini-icons are part of the Silk Icons set of icons at famfamfam.com