MQTT is the quintessential protocol for edge and IoT devices to talk to the cloud for more than a few reasons. If you're a pumpjack in the middle of nowhere west texas 50 miles from town and you're in a field of said pumpjacks getting that data out can be a hassle. Same thing if you're an oil rig off the coast of Mozambique and you *might* have 3g for a *few* hours of the day. Edge compute is starting to grow everywhere. Edge is the 747 cruising over the atlantic. Edge can also be sensors on freezers in grocery stores ensuring that the compressor doesn't die but when it does (cause like datacenters it does at scale) someone is notified before food spoils. Edge is also monitoring the frying temperature in the oil to cook those delicious chicken sandwiches.
The trouble right now is that people have been exposing a full blown linux server on these devices with all the capabilities that linux has and right now edge deployments completely eclipse the 56-57 physical datacenters AWS has and continues to grow unabated. Linux is the all you can eat buffet when what you really want for some of these applications is a cup of miso soup.
To be clear - I LIKE Linux. I've been using linux for a very long time but let's face it - linux is 30 years old and has design concepts that are not congruent with our reality today. It was 'deprecated' the day it came out and was here 10 years before VMWare released GSX and ESX. VMWare is 20 years old now! It also was out 15 years before a small little bookstore in Seattle created "the cloud".
So the developer states - I'll use a container, but there are an awful lot of growing number of people that seem to think container security is even worse than stock linux is. Just yesterday Palo Alto Networks dropped $400M buying TwistLock and earlier this week there was yet another vulnerability in a seemingly endless supply of issues with containers.
So if not linux and not containers what then? How about a unikernel?
Shut up and Show Me the Code
Today I'm going to show you how to run both a set of MQTT clients and a mqtt server in the cloud.
If you're on a mac you can use brew to install mosquitto. We'll do this just so you have ready to use mqtt clients to test but it's not necessary.
brew install mosquittoNow let's install OPS. OPS is a simple tool that will take any linux binary and build a unikernel out of it and allow you to orchestrate them to various cloud services.
curl https://ops.city/get.sh -sSfL | shIf you don't like this way of installing you can always go to the source and build it yourself. OPS has the concept of packages that are very similar to deb packages as we don't expect people to have to build their own commodity software that they don't actually code. I wish we could just re-use debs but they are slightly different.
You can grab the mosquitto package this way. Keep in mind this actually doesn't have the mosquitto client and such in it as it is only one binary with the set of libs and configs it actually needs to run and nothing else. It doesn't even have the support of being able to run the server and the client in the same vm as the kernel doesn't support that - this is for both security and performance. Further if you look at the output you'll see a message about 'running as root'. The truth is that there is no root nor are there non-root users. The only reason you see that message is because we stubbed it out to allow it to run without code modifications. Unlike linux you can't 'root it' and fork a shell. Fork doesn't exist and neither does the shell. You could crash it or you could pull out some ROP gadgets but be forewarned that it has the same ASLR and page protections as Linux does.
Anways, let's quickly test out locally and run it:
➜ t ops pkg load -p 1883 mosquitto_1.5.7
[mosquitto -c /mosquitto.conf]
booting /Users/eyberg/.ops/images/mosquitto.img ...
assigned: 10.0.2.15
1559177870: mosquitto version 1.5.7 starting
1559177870: Config loaded from /mosquitto.conf.
1559177870: Opening ipv4 listen socket on port 1883.
1559177870: Opening ipv6 listen socket on port 1883.
1559177870: Warning: Address family not supported by protocol
1559177870: Warning: Mosquitto should not be run as root/administrator.
1559178137: New connection from 10.0.2.2 on port 1883.
1559178137: New client connected from 10.0.2.2 as mosq/cEjNNo9lqgwxTbu6a0 (c1, k60).
1559178137: Client mosq/cEjNNo9lqgwxTbu6a0 disconnected.
1559178151: New connection from 10.0.2.2 on port 1883.
1559178151: New client connected from 10.0.2.2 as mosq/NALF9bwBJYAvYaHart (c1, k60).
1559178178: New connection from 10.0.2.2 on port 1883.
1559178178: New client connected from 10.0.2.2 as mosq/jaNIW9m5zQrtAJdZC0 (c1, k60).
1559178178: Client mosq/jaNIW9m5zQrtAJdZC0 disconnected.
1559178196: New connection from 10.0.2.2 on port 1883.
1559178196: New client connected from 10.0.2.2 as mosq/2CxLMO1fYmh5LGJ4Kg (c1, k60).
1559178196: Client mosq/2CxLMO1fYmh5LGJ4Kg disconnected.
Looks like it's running and to show you what the end artificat actually is you can see this is a very small disk image. To be clear - this isn't an alpine or busybox type of thing.
➜ ~ ls -lh ~/.ops/images/mosquitto.img
-rw-r--r-- 1 eyberg staff 12M May 29 17:57 /Users/eyberg/.ops/images/mosquitto.img
Wow - that's a big virtual machine - a whopping 12 megabytes - that's not just the mosquitto program - that is everything that is needed to actually run mosquitto.
We can test sending a message to the 'some/topic' topic by starting a subscriber.
➜ ~ mosquitto_sub -h 127.0.0.1 -t some/topic
test
test
and send a message then you'll see the above output:
➜ ~ mosquitto_pub -h 127.0.0.1 -m 'test' -t 'some/topic' -d
Client mosq/2CxLMO1fYmh5LGJ4Kg sending CONNECT
Client mosq/2CxLMO1fYmh5LGJ4Kg received CONNACK (0)
Client mosq/2CxLMO1fYmh5LGJ4Kg sending PUBLISH (d0, q0, r0, m1, 'some/topic', ... (4 bytes))
Client mosq/2CxLMO1fYmh5LGJ4Kg sending DISCONNECT
Build a Mosquitto MQTT Edge Client
Now that we've verified that the mosquitto server is listening locally we can craft a Go client to talk to it. We'll use this as our 'edge' client.
You'll want to go get this mqtt client first. If you're not a gopher you can just follow along with the mosquitto_sub tool you installed earlier with brew.
package main
import (
"fmt"
mqtt "github.com/eclipse/paho.mqtt.golang"
"os"
)
func main() {
topic := "some/topic"
host := os.Getenv("MHOST")
opts := mqtt.NewClientOptions().AddBroker("tcp://" + host + ":1883")
client := mqtt.NewClient(opts)
if token := client.Connect(); token.Wait() && token.Error() != nil {
fmt.Println(token.Error())
os.Exit(1)
}
token := client.Publish(topic, 0, false, "some pig")
token.Wait()
if token.Error() != nil {
fmt.Println(token.Error())
os.Exit(1)
}
}
We'll build this as a native linux binary:
GOOS=linux go buildThen we'll set a environment variable for this in config.json:
{
"Env": {
"MHOST": "127.0.0.1"
}
}
Environment variables will be injected into your disk image at run-time.
You should be able to test it against your local server:
ops run -c config.js gom
Finding dependent shared libs
booting /Users/eyberg/.ops/images/gom.img ...
assigned: 10.0.2.15
Woot! - ok - on to the next step.
Deploy To Google Cloud
Ok - enough of show and tell - let's deploy this bad boy to google cloud. If you don't have a Google Cloud account you'll need to sign up for one to continue this adventure (You're welcome Thomas.) .
You'll also need to grab a service worker key and create a cloud bucket to store the disk images.
Once you get the service worker key put put it in a conveinent place (I used to practice $HOME cleanliness but it's basically trash these days) and export it:
GOOGLE_APPLICATION_CREDENTIALS=~/gcloud.jsonAlso we'll want a json file to put some cloud config into:
Just put in your project_id, preferred zone and the cloud bucket you created earlier.
{
"CloudConfig" :{
"ProjectID" :"some-project-1234",
"Zone": "us-west2-a",
"BucketName":"my-special-bucket"
},
"RunConfig" : {
"Memory": "2G"
}
}
Now let's create that instance:
➜ mserver ops image create -c config.json -p mosquitto_1.5.7
[mosquitto -c /mosquitto.conf]
bucket found: nanos-deploy
Image creation started. Monitoring operation
operation-1559234952285-58a1dad73cc99-59a4d9dc-b690066f.
............
Operation operation-1559234952285-58a1dad73cc99-59a4d9dc-b690066f
completed successfullly.
Image creation succeeded nanos-mosquitto-image.
gcp image 'nanos-mosquitto-image' created...
Keep in mind that we aren't spinning up any instances yet - we are just creating the virtual machine - the equivalent of this is running a terraform job except you aren't configuring a linux box - you're configuring mosquitto to run by itself.
You should see something like this after:
➜ mserver ops image list
+-----------------------+--------+-------------------------------+
| NAME | STATUS | CREATED |
+-----------------------+--------+-------------------------------+
| nanos-mosquitto-image | READY | 2019-05-30T09:49:12.940-07:00 |
+-----------------------+--------+-------------------------------+
Now we can spin it up like this:
➜ mserver ops instance create nanos-mosquitto-image -p my-project-1234 -z us-west2-a
Instance creation started using image
projects/my-project-1234/global/images/nanos-mosquitto-image. Monitoring
operation operation-1559235183199-58a1dbb3746d6-793cd05a-843fce3c.
.....
Operation operation-1559235183199-58a1dbb3746d6-793cd05a-843fce3c
completed successfullly.
Instance creation succeeded nanos-mosquitto-image-1559235182.
now we should see it running:
➜ mserver ops instance list -z us-west2-a -p my-project-1234
+----------------------------------+---------+-------------------------------+-------------+---------------+
| NAME | STATUS | CREATED
| PRIVATE IPS | PUBLIC IPS |
+----------------------------------+---------+-------------------------------+-------------+---------------+
| nanos-mosquitto-image-1559235182 | RUNNING | 2019-05-30T09:53:04.612-07:00 | 10.240.0.51 | 35.236.41.105 |
+----------------------------------+---------+-------------------------------+-------------+---------------+
Now mqtt by default listens on port 1883 - there's a half dozen ways to skin the cat when it comes to networking but in this case we can add a firewall rule tag that allows 1883 when using the 'mqtt' tag.
Let's test it out:
First subscribe to some topic:
➜ ~ mosquitto_sub -h 35.236.41.105 -t 'some/topic'
test
Then we can publish a message:
➜ mserver mosquitto_pub -h 35.236.41.105 -m 'test' -t 'some/topic' -d
Client mosq/3uNeYjlQqeTNHMMnfy sending CONNECT
Client mosq/3uNeYjlQqeTNHMMnfy received CONNACK (0)
Client mosq/3uNeYjlQqeTNHMMnfy sending PUBLISH (d0, q0, r0, m1, 'some/topic', ... (4 bytes))
Client mosq/3uNeYjlQqeTNHMMnfy sending DISCONNECT
Cool! Remember that go client we had earlier? Let's try that out inside a unikernel.
First let's cahnge the ip:
{
"Env": {
"MHOST": "35.236.41.105"
}
}
Now we can run it:
➜ gom ops run -c config.js gom
Finding dependent shared libs
booting /Users/eyberg/.ops/images/gom.img ...
assigned: 10.0.2.15
exit status 1
.. and if you still have your subscription open you can see the subscriber got the message!
➜ ~ mosquitto_sub -h 35.236.41.105 -t 'some/topic'
some pig
Now since this is a unikernel we don't have ssh listening on this cause that would imply that we have some sort of interactive access to run arbitrary programs on this instance which we absolutely do not have the capability for - as in the kernel bits don't even have the capabilities to do so - on purpose. That's ok though cause if we want the logs we can grab them this way or log them to papertrail or splunk or something (the ips have been changed to protect the innocent):
➜ gom ops instance logs nanos-mosquitto-image-1559235182 -p my-project-1234
-z us-west2-a
SeaBIOS (version 1.8.2-20190503_170316-google)
Total RAM Size = 0x0000000080000000 = 2048 MiB
CPUs found: 1 Max CPUs supported: 1
found virtio-scsi at 0:3
virtio-scsi vendor='Google' product='PersistentDisk' rev='1' type=0
removable=0
virtio-scsi blksize=512 sectors=2097152 = 1024 MiB
drive 0x000f27c0: PCHS=0/0/0 translation=lba LCHS=1024/32/63 s=2097152
Booting from Hard Disk 0...
assigned: 10.240.0.51
1559235191: mosquitto version 1.5.7 starting
1559235191: Config loaded from /mosquitto.conf.
1559235191: Opening ipv4 listen socket on port 1883.
1559235191: Opening ipv6 listen socket on port 1883.
1559235191: Warning: Address family not supported by protocol
1559235191: Warning: Mosquitto should not be run as root/administrator.
1559235769: New connection from 1.2.3.4 on port 1883.
1559235769: New client connected from 1.2.3.4 as mosq/rpMpk5txbXh4g9Dcok (c1, k60).
1559235773: New connection from 1.2.3.4 on port 1883.
1559235773: New client connected from 1.2.3.4 as mosq/3uNeYjlQqeTNHMMnfy (c1, k60).
1559235773: Client mosq/3uNeYjlQqeTNHMMnfy disconnected.
1559235863: Socket error on client mosq/rpMpk5txbXh4g9Dcok, disconnecting.
1559235936: New connection from 1.2.3.4 on port 1883.
1559235936: New client connected from 1.2.3.4 as mosq/Z4Z6rQqQuvjhCGNHGq (c1, k60).
1559235949: New connection from 1.2.3.4 on port 1883.
1559235949: New client connected from 1.2.3.4 as 7171972b-36b2-4181-a54e-ea69c65a621e (c1, k30).
1559235949: Socket error on client 7171972b-36b2-4181-a54e-ea69c65a621e,
disconnecting.
➜ ~ ops instance delete -p my-project-1234 -z us-west2-a
nanos-mosquitto-image-1559235182
Instance deletion started. Monitoring operation
operation-1559238749116-58a1e8fc2d9e6-e8ad5172-0d0e268c.
............................................................
Operation operation-1559238749116-58a1e8fc2d9e6-e8ad5172-0d0e268c
completed successfullly.
Instance deletion succeeded nanos-mosquitto-image-1559235182.
➜ ~ ops image delete -i nanos-mosquitto-image
..
Operation operation-1559238979748-58a1e9d82021e-c49229c9-c6a141cb
completed successfullly.
Image deletion succeeded nanos-mosquitto-image.
Well - I hope you find this interesting and as you can tell OPS is open source and has lots of room for new and interesting ways of deploying these things so if you like what you see feel free to jump into the source at https://github.com/nanovms/ops open up some pulls.
