InfluxDB is a popular time series database that fits well inside an edge or IoT pipeline where you might be already using something like MQTT. If you're already using unikernels for your edge devices it might be a good idea to turn your time series datastore into one as well.
If you're new to this series of tutorials first things first - go get OPS the unikernel builder/orchestrator.
Now we can download a pre-made package for InfluxDB:
➜ ~ ops pkg get influxdb_1.7.0
Downloading..
https://storage.googleapis.com/packagehub/influxdb_1.7.0.tar.gz
15.67 MiB / 15.67 MiB
[=============================================================================================================================================================]
100.00% 3.82 MiB/s 4s
You can see that the entire thing is only 16 meg zipped - that's the entire disk image. Unlike a container this has everything you need to run not just the application. Also, unlike a container the end artifact is a virtual machine meaning you can do all the wonderful things like live migration that containers still can not do properly.
To deploy this as a unikernel on Google Cloud you'll need two things from Google. One is a service key. The other is a a cloud bucket to store the disk images in.
Then we'll export our service key like so:
export GOOGLE_APPLICATION_CREDENTIALS=~/gcloud.json{
"CloudConfig" :{
"ProjectID" :"my-project",
"Zone": "us-west2-a",
"BucketName":"my-bucket"
},
"RunConfig" : {"Memory": "2G"}
}
From there we can create a new disk image for google. If you've ever used something like terraform before this is doing the same sort of task except it's not creating a linux instance. It's turning your application, in this case, InfluxDB into it's own little instance. This is different from an alpine or coreos or any other JEOS (just enough operating system) because it is custom built only to run one application - faster and safer than linux. For instance there is no ssh listening nor is there a shell available you can pop. Further unlike these less safe environments there is no capability of running arbitrary 3rd party programs like cryptominers.
➜ itest ops image create -c config.json -p influxdb_1.7.0
[influxd]
bucket found: my-bucket
Image creation started. Monitoring operation
operation-1561006967942-58bba41d0eccb-360ca533-2d4645be.
...............
Operation operation-1561006967942-58bba41d0eccb-360ca533-2d4645be
completed successfullly.
Image creation succeeded nanos-influxd-image.
gcp image 'nanos-influxd-image' created...
Once we have the image built we can now spin an instance of it up:
➜ itest ops instance create nanos-influxd-image -p my-project -z us-west2-a
Instance creation started using image
projects/my-project/global/images/nanos-influxd-image. Monitoring
operation operation-1561007316022-58bba569033e6-0780e624-ce8f46ca.
.....
Operation operation-1561007316022-58bba569033e6-0780e624-ce8f46ca
completed successfullly.
Instance creation succeeded nanos-influxd-image-1561007315.
Let's go ahead and create a firewall rule to allow port 8086 to talk:
Note: This is merely a quick tutorial and we would not recommend exposing the database directly to the public like this without at least authentication, however it probably just belongs in it's own private network.
If you don't have the ip you can obtain it from the 'instances list' command. You can also see the image name this way:
➜ ~ ops instance list -p my-project -z us-west2-a
+--------------------------------+---------+-------------------------------+-------------+---------------+
| NAME | STATUS | CREATED
| PRIVATE IPS | PUBLIC IPS |
+--------------------------------+---------+-------------------------------+-------------+---------------+
| nanos-influxd-image-1561007315 | RUNNING | 2019-06-19T22:08:37.099-07:00 | 10.240.0.58 | 34.94.231.104 |
+--------------------------------+---------+-------------------------------+-------------+---------------+
Now that that is done we can run the influx client and talk to our new database (gratuitously ripped from the influx getting started guide).
me@box:~/i/influxdb-1.7.0~n201901080800-0/usr/bin$ ./influx -host 34.94.231.104
Connected to http://34.94.231.104:8086 version 1.7.0~n201901080800
InfluxDB shell version: 1.7.0~n201901080800
Enter an InfluxQL query
> create database somepig
> show databases
name: databases
name
----
_internal
somepig
> use somepig
Using database somepig
> INSERT cpu,host=serverA,region=us_west value=0.64
> INSERT cpu,host=serverA,region=us_west value=0.64
> INSERT cpu,host=serverA,region=us_west value=0.64
> INSERT cpu,host=serverA,region=us_west value=0.64
> SELECT "host", "region", "value" FROM "cpu"
name: cpu
time host region value
---- ---- ------ -----
1561007976938723000 serverA us_west 0.64
1561007978835553000 serverA us_west 0.64
1561007979359785000 serverA us_west 0.64
1561007980399038000 serverA us_west 0.64
Want to see the server side logs? We can grab logs this way but feel free to ship them to papertrail or splunk or any other logging system that you might be already using.
➜ ~ ops instance logs -p my-project -z us-west2-a
nanos-influxd-image-1561007315
SeaBIOS (version 1.8.2-20190503_170316-google)
Total RAM Size = 0x000000006cc00000 = 1740 MiB
CPUs found: 1 Max CPUs supported: 1
found virtio-scsi at 0:3
virtio-scsi vendor='Google' product='PersistentDisk' rev='1' type=0
removable=0
virtio-scsi blksize=512 sectors=2097152 = 1024 MiB
drive 0x000f27c0: PCHS=0/0/0 translation=lba LCHS=1024/32/63 s=2097152
Booting from Hard Disk 0...
assigned: 10.240.0.58
ts=2019-06-20T05:08:43.195633Z lvl=info msg="InfluxDB starting" log_id=0G8pRSEl000 version=1.7.0~n201901080800 branch=master commit=4d21ab9e0e39a9ef89e4723da789ebf144350d78
ts=2019-06-20T05:08:43.198184Z lvl=info msg="Go runtime" log_id=0G8pRSEl000 version=go1.11 maxprocs=1
ts=2019-06-20T05:08:43.329462Z lvl=info msg="Using data dir" log_id=0G8pRSEl000 service=store path=/root/.influxdb/data
ts=2019-06-20T05:08:43.331539Z lvl=info msg="Compaction settings" log_id=0G8pRSEl000 service=store max_concurrent_compactions=1
throughput_bytes_per_second=50331648
throughput_bytes_per_second_burst=50331648
ts=2019-06-20T05:08:43.335660Z lvl=info msg="Open store (start)" log_id=0G8pRSEl000 service=store trace_id=0G8pRSml000 op_name=tsdb_open op_event=start
ts=2019-06-20T05:08:43.337921Z lvl=info msg="Open store (end)" log_id=0G8pRSEl000 service=store trace_id=0G8pRSml000 op_name=tsdb_open op_event=end op_elapsed=2.267ms
ts=2019-06-20T05:08:43.340426Z lvl=info msg="Opened service" log_id=0G8pRSEl000 service=subscriber
ts=2019-06-20T05:08:43.341577Z lvl=info msg="Starting monitor service" log_id=0G8pRSEl000 service=monitor
ts=2019-06-20T05:08:43.343015Z lvl=info msg="Registered diagnostics client" log_id=0G8pRSEl000 service=monitor name=build
ts=2019-06-20T05:08:43.344878Z lvl=info msg="Registered diagnostics client" log_id=0G8pRSEl000 service=monitor name=runtime
ts=2019-06-20T05:08:43.347361Z lvl=info msg="Registered diagnostics client" log_id=0G8pRSEl000 service=monitor name=network
ts=2019-06-20T05:08:43.349059Z lvl=info msg="Registered diagnostics client" log_id=0G8pRSEl000 service=monitor name=system
ts=2019-06-20T05:08:43.351604Z lvl=info msg="Starting precreation service" log_id=0G8pRSEl000 service=shard-precreation check_interval=10m advance_period=30m
ts=2019-06-20T05:08:43.354151Z lvl=info msg="Starting snapshot service" log_id=0G8pRSEl000 service=snapshot
ts=2019-06-20T05:08:43.355546Z lvl=info msg="Starting continuous query service" log_id=0G8pRSEl000 service=continuous_querier
ts=2019-06-20T05:08:43.357149Z lvl=info msg="Starting HTTP service" log_id=0G8pRSEl000 service=httpd authentication=false
ts=2019-06-20T05:08:43.359024Z lvl=info msg="opened HTTP access log" log_id=0G8pRSEl000 service=httpd path=stderr
ts=2019-06-20T05:08:43.360761Z lvl=info msg="Listening on HTTP" log_id=0G8pRSEl000 service=httpd addr=0.0.0.0:8086 https=false
ts=2019-06-20T05:08:43.362620Z lvl=info msg="Starting retention policy enforcement service" log_id=0G8pRSEl000 service=retention check_interval=30m
ts=2019-06-20T05:08:43.365324Z lvl=info msg="Listening for signals" log_id=0G8pRSEl000
ts=2019-06-20T05:08:43.367060Z lvl=info msg="Storing statistics" log_id=0G8pRSEl000 service=monitor db_instance=_internal db_rp=monitor interval=10s
ts=2019-06-20T05:08:43.368954Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0G8pRSEl000
[httpd] 50.197.181.22 - - [20/Jun/2019:05:18:39 +0000] "GET /ping HTTP/1.1" 204 0 "-" "InfluxDBShell/1.7.0~n201901080800" dccbfd16-931a-11e9-8001-f429ff970983 132
ts=2019-06-20T05:19:19.433874Z lvl=info msg="Executing query" log_id=0G8pRSEl000 service=query query="CREATE DATABASE somepig"
[httpd] 50.197.181.22 - - [20/Jun/2019:05:19:19 +0000] "POST /query?chunked=true&db=&epoch=ns&q=create+database+somepig HTTP/1.1" 200 57 "-" "InfluxDBShell/1.7.0~n201901080800" f4a40d70-931a-11e9-8002-f429ff970983 21398
ts=2019-06-20T05:19:25.520203Z lvl=info msg="Executing query" log_id=0G8pRSEl000 service=query query="SHOW DATABASES"
[httpd] 50.197.181.22 - - [20/Jun/2019:05:19:25 +0000] "POST /query?chunked=true&db=&epoch=ns&q=show+databases HTTP/1.1" 200 118 "-" "InfluxDBShell/1.7.0~n201901080800" f844c05a-931a-11e9-8003-f429ff970983 11475
ts=2019-06-20T05:19:30.152204Z lvl=info msg="Executing query" log_id=0G8pRSEl000 service=query query="SHOW DATABASES"
[httpd] 50.197.181.22 - - [20/Jun/2019:05:19:30 +0000] "POST /query?db=&epoch=ns&q=SHOW+DATABASES HTTP/1.1" 200 118 "-" "InfluxDBShell/1.7.0~n201901080800" fb078a48-931a-11e9-8004-f429ff970983 12895
[httpd] 50.197.181.22 - - [20/Jun/2019:05:19:36 +0000] "POST /write?consistency=all&db=somepig&precision=ns&rp= HTTP/1.1" 204 0 "-" "InfluxDBShell/1.7.0~n201901080800" ff131b02-931a-11e9-8005-f429ff97098 115572
[httpd] 50.197.181.22 - - [20/Jun/2019:05:19:38 +0000] "POST /write?consistency=all&db=somepig&precision=ns&rp= HTTP/1.1" 204 0 "-" "InfluxDBShell/1.7.0~n201901080800" 00348d4a-931b-11e9-8006-f429ff970983 3224
[httpd] 50.197.181.22 - - [20/Jun/2019:05:19:39 +0000] "POST /write?consistency=all&db=somepig&precision=ns&rp= HTTP/1.1" 204 0 "-" "InfluxDBShell/1.7.0~n201901080800" 00848aca-931b-11e9-8007-f429ff970983 2872
[httpd] 50.197.181.22 - - [20/Jun/2019:05:19:40 +0000] "POST /write?consistency=all&db=somepig&precision=ns&rp= HTTP/1.1" 204 0 "-" "InfluxDBShell/1.7.0~n201901080800" 01231ece-931b-11e9-8008-f429ff970983 3504
ts=2019-06-20T05:19:48.320399Z lvl=info msg="Executing query" log_id=0G8pRSEl000 service=query query="SELECT host, region, value FROM somepig.autogen.cpu"
[httpd] 50.197.181.22 - - [20/Jun/2019:05:19:48 +0000] "POST /query?chunked=true&db=somepig&epoch=ns&q=SELECT+%22host%22%2C+%22region%22%2C+%22value%22+FROM+%22cpu%22 HTTP/1.1" 200 177 "-" "InfluxDBShell/1.7.0~n201901080800" 05dbc79a-931b-11e9-8009-f429ff970983 15386
Once you're done playing around you can spin down your instance this way:
➜ ~ ops instance delete -p my-project -z us-west2-a
nanos-influxd-image-1561007315
Instance deletion started. Monitoring operation
operation-1561008320756-58bba927339a5-a7bc4272-7bf23fff.
..........................................................
Operation operation-1561008320756-58bba927339a5-a7bc4272-7bf23fff
completed successfullly.
Instance deletion succeeded nanos-influxd-image-1561007315.
Then you can finally delete your image like this:
➜ ~ ops image delete -i nanos-influxd-image
..
Operation operation-1561008650597-58bbaa61c35bd-a74068c5-9228dab6
completed successfullly.
Image deletion succeeded nanos-influxd-image.
Hope you found this way of deploying software a *lot* easier than certain methods out there. In a world of ever-increasing complexity creep into the devops pipeline unikernels are a refreshing breath of clean crisp air.
