OpenStack: Fix Security Group Enforcement on Compute Nodes

I discovered the instances in my home lab were not being protected by the neutron security groups I’d set up for them – what followed was a week-long odyssey to discover the culprit. IPTables rules were being put in place, and every other aspect of Neutron networking was just fine. Making things more mysterious, a test deploy on my own desktop, with the host running Fedora 23 and the containers running CentOS 7, did not manifest the issue.

A handy diagram from the excellent “OpenStack Hackspace” – click image to check it out.

Security groups are applied to the tap interface closest to the virtual machine, shown above (not mine – click the image to see the rest of that excellent tutorial).

It turns out there’s a pair of kernel tunables that govern whether or not iptables rules are applied to  interfaces that are members of a bridge, as in this instance. It also turns out that Red Hat, for some reason, toggled the default value of this tunable to 0 after RHEL 7 was released.

These tunables, and their required values, are:

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

And this needs to be set on the outer host, if you run your OpenStack cloud in containers. If not and/or otherwise, you need this set on your compute nodes as this is where security groups are set.

Hope this saves someone a week of troubleshooting! 🙂

OpenStack: Dedicate Compute Hosts to Projects

Use case: We want to deploy instances on a particular set of compute hosts because of their special or specialized capabilities.

On the API server(s) ensure that the following scheduler_default_filters are set: AggregateInstanceExtraSpecsFilter,AggregateMultiTenancyIsolation

Dedicate Compute Hosts To a Specific Project

Create a host aggregate:

$ nova aggregate-create DedicatedCompute
$ nova aggregate-add-host DedicatedCompute dedicated-compute-host1
$ nova aggregate-add-host DedicatedCompute dedicated-compute-host2
$ nova aggregate-add-host DedicatedCompute dedicated-compute-host3

Add a key=value pair to the host aggregate metadata – we’ll use this to match against later.

$ nova aggregate-set-metadata DedicatedCompute filter_tenant_id=<Tenant ID Here>

We are here using the AggregateMultiTenancyIsolation filter. If we stop here, only members of the specified tenant will be able to create instances on hosts in this aggregate – but instances will also spawn on any other host that is either not in an aggregate, or has no filter_tenant_id metadata set. We want to isolate these hosts to a specific project.

Isolate Hosts To a Specific Project

We do so by Creating a flavor and giving it specific metadata:

$ nova flavor-key m1.dedicated set aggregate_instance_extra_specs:filter_tenant_id=<Tenant ID Here>

We are here invoking the AggregateInstanceExtraSpecsFilter filter. Note a couple of things:

  1. We’re filtering on the filter_tenant_id= tag we applied to the host aggregate above. This is a convenience – we could have set another arbitrary key=value pair in the host aggregate’s metadata and used that to match against here. This is conceptually important for the purpose of understanding how the two filters work – they don’t work together, we just happen to be using the same tags.
  2. The format of the above is very important. If you specify this in any other form the ComputeCapabilitiesFilter will try to match the resultant tag to a host and fail to start an instance with that flavor. This can make troubleshooting interesting – I had to walk through the code path of the nova scheduler and the filters to find this out. Fun!

Isolate Storage To a Specific Project

In this project’s case we want a specific storage pool, itself dedicated to a specific set of hosts and disks, available for use by instances in this project – but not other projects. We have created a volume backend called ‘elasticsearch’ that points to this storage pool, and will now create a Cinder volume type that makes use of it.

$ cinder type-create dedicated
$ cinder type-key dedicated set volume_backend_name=dedicated

We start by ensuring that all other projects will not be able to use this volume type:

$ for project in `openstack project list -f value | awk '{print $1}'`; do cinder quota-update --volumes 0 --volume-type=dedicated $project; done

We then grant a quota for this specific volume type to our special project:

$ cinder quota-update --volumes 100 --volume-type dedicated <Tenant ID Here>



Importing an OpenStack VM into Amazon EC2

Some quick notes for those interested:

  • Install EC2 API TOOLS
  • Set Access and Secret key environment variables:
    • export AWS_ACCESS_KEY=xxxxxxxxxxxxxxx
    • export AWS_SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  • Set up an S3 bucket and secure it to taste, to be specified later
  • If exporting from OpenStack, remove the cloud-init package
  • Note all prep considerations on
  • Export the VM image – I’m using OpenStack (KVM) images in raw format, so these needed no initial conversion. You can also use VHD or VMDK
    • Docs:
    • Command I used: ec2-import-instance ./my-awesome-image.img -f RAW -t c4.xlarge -a x86_64 -b my-awesome-bucket –region us-east-1 –availability-zone us-east-1a –subnet subnet-xxxxxx -o ${AWS_ACCESS_KEY} -w ${AWS_SECRET_KEY} -p Linux

You need to take the destination region into account – your subnet and S3 bucket be available to the region/availability zone you specify above.

The import takes time after the upload is complete – watch progress with ec2-describe-conversion-tasks.

Privacy Helper for Windows 10

During my evaluation of Windows 10 I’ve cobbled together a script that disables most known anti-privacy features in the new system. It also removes the unnecessarily installed default Modern apps. The script was designed with a professional environment in mind, but applies equally well to home users, and most settings will apply to all editions of Windows, not just Pro/Enterprise

It’s alarming but not surprising that we’ve gotten to this point – Windows has always served the agenda of its maker before that of its user. This edition ships with a significant set of end user facing UI improvements, but its goals are not different. They simply take advantage of the current state of the art and the current anti-privacy climate.

The script can’t be considered complete as not all anti-privacy features are documented by Microsoft, and future updates will doubtless add more.  list of changes – comment each section of the script out if one of these is undesired:

  • Stops / Disables intrusive diagnostics services
  • Removes diagnostic scheduled tasks
  • Removes OneDrive
  • Removes access to the Windows Store
  • Removes access to Cortana
  • Block connection to Microsoft Accounts
  • Disables WiFi Sense (what’s this?)
  • Disables Windows Update peer to peer (what’s this?)
  • Requires Ctrl-Alt-Del to log on
  • Blocks “Add features to Windows 10”
  • Removes unnecessary apps (they can be reinstalled from the Store if desired, assuming you’ve left it enabled)

Standalone – an extension for Mozilla Firefox

Found this little gem today, thought I’d mention it on my corner of the web.

Creates site specific apps, like Prism used to do – for Chrome users, this is the equivalent of “Create Application Shortcuts” which I find indispensable personally.

The strange thing is, it’s been around since 2013. How did I miss it for that long? I’ve been looking everywhere for something like this.

Thoughts on Docker

I like the concept of Docker and containerization in general, but I have some pretty fundamental concerns:

Thought experiments:

  • How many deployed docker images were torn down and redeployed upon the revelation of heartbleed? Of shellshock? In practice, not in theory.
  • How many Docker images are regularly destroyed and redeployed for the purpose of updating their userlands? Again, in reality, even with the most agile orchestration.
  • How many Docker images are actually deployed with a minimal attack surface, that being only the executables and libraries they need, rather than entire userlands?
  • How many Docker images are given to IT/Ops as single filesystem images rather than multi-gigabyte change layers, contributing heavily to wasted storage space?
  • How can Docker images composed of random people’s aging Linux userlands ever be taken seriously in an environment that needs to be kept certified, stable and secure?
  • What is the benefit of Docker given the above, when LXC and Libvirt-LXC performs the same containerization and provides Ops with much greater flexibility in terms of orchestration and change management, and has for years?
  • Dan Walsh of Red Hat has much to say about the security of Docker and LXC containers – the most important statement he makes is that “containers don’t contain” – containers provide no security, they are only useful for the purpose of deploying applications in a manageable way. Given this, is it responsible to use containers based on full Linux filesystems? If you do, you’d better be ready to tear down your ENTIRE stack each and every time a major vulnerability comes to light.

Points worth pondering – these affect the future direction of container technology and shed light on the implications.

Running Systemd within a Docker image

NOTE: This is not for general purpose use – CAP_SYS_ADMIN grants the container a large number of dangerous privileges – this should be used only by a sysadmin (Ops) – not a developer – for the purpose of containerizing infrastructure.

Some of my insights gained on running systemd within docker – I’m aware the general idea is to run a single process, but that’s for developers. I’m a sysadmin, so I know that underlying “docker” as a management system is a full featured Linux namespace/cgroup facility allowing me to run a fully containerized Linux userland.

This is cobbled together from some of the example Dockerfiles provided by Red Hat’s Project Atomic as well as personal research into systemd, unprivileged container limitations, and the Linux capability system. This isn’t guaranteed to work on anything but the latest version of Docker at the time of this post – 1.4.1. May work on a previous version if it has support for –add-cap.

FROM centos:centos7
MAINTAINER "Brad Laue" <>
RUN yum -y update; yum clean all \
yum -y swap -- remove fakesystemd -- install systemd systemd-libs \
(cd /lib/systemd/system/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/*; \
rm -f /lib/systemd/system/*udev*; \
rm -f /lib/systemd/system/*initctl*; \
rm -f /lib/systemd/system/*;\
rm -f /lib/systemd/system/*;
RUN systemctl mask
RUN systemctl disable; systemctl enable
VOLUME ["/sys/fs/cgroup"]
VOLUME ["/run"]
VOLUME ["/export"]
ENV container=lxc
CMD ["/usr/lib/systemd/systemd"]

Build with:

docker build --rm -t bradlaue/centos-systemd .

Run with:

docker run -ti --cap-add=SYS_ADMIN -e container=lxc bradlaue/centos-systemd

Note this does not require a fully privileged container instance as many seem to indicate – systemd requires only CAP_SYS_ADMIN in order to avoid a segfault.

From here you can build out a container by installing/running standard Linux services in the normal systemd way, including process monitoring.

Pew Research Report – Net Threats

“There are too many institutional players interested in restricting, controlling, and directing ‘ordinary’ people’s ability to make, access, and share knowledge and creative works online — intellectual property rights holders, law enforcement and security agencies, religious and cultural censors, political movements and parties, etc. For a long time I’ve felt that the utopianism, libertarianism, and sheer technological skill of both professional and amateur programmers and engineers would remain the strongest counterbalance to these restrictive institutional pressures, but I’m increasingly unsure as the technologists themselves and their skills are being increasingly restricted, marginalized, and even criminalized.”

In A Battle For Web Traffic, Bad Bots Are Going After Grandma : All Tech Considered : NPR

If bad bots were left to their own devices, bad taste would dominate the Web. But, even worse, Kaminsky worries that this kind of advertising fraud is undermining the economics of the Web.

Though people “tend not to like advertisers, advertisers have paid for a network that allows greater interpersonal communication than any other time in history. Who paid for all this free service? They did,” he says.

Kaminsky’s firm works with advertises to fight this problem. He worries that advertisers will go back to the TV or other outlets, which are better protected against fraud.

I honestly wish they would. Such a move would radically change the current state of affairs online for the better. Profit seeking is not a solid modus operandi for building a communication / educational network that can truly benefit people.

Nearly one-third of Americans aren’t ready for the next generation of technology | Science/AAAS

A new survey suggests that the digital divide has been replaced by a gap in digital readiness. It found that nearly 30% of Americans either aren’t digitally literate or don’t trust the Internet. That subgroup tended to be less educated, poorer, and older than the average American.

I would content that the greatest level of technical literacy inspires by far the greatest mistrust in the Internet. Held together by little bits of string and the lies marketing departments try to tell everyone, those with more age / experience are wise to think twice about depending on the Internet to manage their personal lives.

Doing so is a fools errand that younger people haven’t thought through, as is the case with so many other things in their lives.