Motherboard: Intel Desktop Board DH55TC
CPU: Intel Core I3 530 2,93Ghz
Memory: OCZ Gold Low Voltage OCZ3G1333LV4GK (8GB)
Harddisk 1: Kingston SSDNow SNV425-S2BN/64GB
Harddisk 2: 500GB Western Digital Green

All of the hardware is housed in a small desktop enclosure (as shown on the picture)

Hypervisor/Virtualization
With the initial setup of my home server I started with a Windows 2008 R2 installation, with Hyper-V on top.
After running on this system for about 2 months, I really couldn’t overcome the lack of USB passthrough in Hyper-V. A lot of domotica interfaces use USB, so I couldn’t use them within virtual machines.
I could off course still run my home automation software on the host (Windows 2008 R2), but this has a downside on the flexibility side. I just want my home automation software within a VM so it can be easily moved around to different hardware in case my server crashes.
This is where VMware ESXi comes to the rescue! ESXi does offer USB passthrough, it works very convenient. You can just plugin USB devices to the host and then add them to a Virtual Machine as hardware.
Here’s an example of a few interfaces connected to my home automation VM:

VMware ESXi (5.0) is offered for free with a few limitations:

- You can use a maximum of 32GB
- You can use only one processor (with unlimited amount of cores)

Both limitations are not a big problem for my home server setup. More information about VMware ESXi editions can be found here.

Virtual machines

I run quite a few VM’s on my home server, here’s a list:

- One Linux (Debian 6.0) machine running my home automation software (HouseAgent).
- One OSX Lion machine, for testing purposes (basically to test HouseAgent on OSX)
- Two Windows servers for general purpose (file sharing etc.)
- One Windows 7 Virtual Machine for general purpose and testing
- I also have some VM’s that are in the “Off” state but are sometimes powered on for testing, such a System Center Operations Manager (SCOM) test environment.

I hear you asking, OSX on ESXi? But isn’t that unsupported? Yes, it’s indeed unsupported but thanks to the ESXi unlocker for OSX this is now possible. The unlocker can be downloaded here. It runs very smooth with VMware tools installed.

So there you have it: three platforms (Linux, Windows and OSX) on one single server! Something I was looking for, for quite a while..

Disk layout
All VM’s use more or less the same disk layout. The OS disks are created on the SSD disk, whereas the data disks are created on the Western Digital disk.
My home automation is entirely created on the SSD, for optimal performance.

Performance
So what about performance? Well, even with 5 VM’s located on this hardware it still runs very smooth. Most of the time the VM’s are idle anyway.
CPU usage is around 15% on average. Memory usage is 7GB out of 8GB.

This concludes the first part of these blog series, I hope you enjoyed it so far. In the next post we’ll dive into the backup methods I use for my home server.

I think his/her spambot had to a pick a random word from those lists, but somehow it failed :-)
So, here’s a lesson.. if you do things like this, do them right and properly!

- 8Kb flash (yes, space does matter!)
- 0.5Kb SRAM
- Maximum operating frequency (20Mhz, using an external crystal)

Here’s a picture of this MCU:

To program this MCU you have to use an ISP (In System Programmer) programmer. I own an USBTinyISP produced by Adafruit, which is perfect for this job. There is a minor drawback though: the ATtiny chips are not supported out of the box within the Arduino IDE. However, there is a perfect open source library by the name of arduino-tiny available to support this MCU within the Arduino IDE.

Here’s a picture of my test setup with the USBTinyISP (on the right) connected:

The pins are connected as follows:

• ATtiny Pin 2 to USBTinyISP SCK
• ATtiny Pin 1 to USBTinyISP MISO
• ATtiny Pin 0 to USBTinyISP MOSI
• ATtiny RST pin to USBTinyISP RESET

After getting the connections right, I started my first programming attempt. The test sketch blink, which is the “hello world” for MCU’s compiled correctly but then:

Hmm, nasty java exception there. After some digging around I figured out the incorrect programmer was getting addressed in the boards.txt file coming with the arduino-tiny project. This file is located under in the Arduino hardware/tiny/ folder. In order to make it work with the USBTinyISP, you have to change the following line for each target board (the ATtiny85 at 1mhz is illustrated here):

attiny85at1.upload.using=pololu

Into the following:

attiny85at1.upload.using=arduino:usbtinyisp

After restarting the IDE, I was able to program the sketch!

Debugging

For standard Arduino’s, you can use a built in serial interface to debug. There is no such thing available for ATtiny chips. However, the arduino-tiny library does offer a debug interface. I used the USB-BUB from Modern Device to setup a debug interface. For the ATtiny85 the standard arduino-tiny debug interface is tied to PB3 of the ATtiny.
Once setup, you can use the regular Serial.begin() and Serial.print/println() functions. I connected PB3 to the RX pin of the USB-bub, off course there’s also a ground connection required so I connected the GND of the USB-BUB to the GND of my ATtiny. After that I was ready to start debugging…

So, here we go.. ready to start tinkering!

Looking for alternatives ZeroMQ caught my eye. A major difference is that ZeroMQ uses a brokerless design whereas RabbitMQ uses a broker based design.
Let’s have a look at both designs:

As you can see, there is a major change in architecture. However because we use a central “coordinator” already, that coordinator took over the role of “broker” role as well.
This way the change to ZeroMQ has minimal impact on plugins, or the way HouseAgent works in general. This architecture leaves some space for future high availability/failover scenarios, by implementing a second “coordinator”. I don’t want this post to become a RabbitMQ vs ZeroMQ comparison (just Google if you are looking for that), but I did want to point out this essential change.

So let’s look at the size of ZeroMQ. The size of ZeroMQ is a lot less (being c/c++), the shared library is 1.6MB and can probably be stripped down quite a bit by removing unused features. 70MB-1.6MB = 68,4MB space saving!

The code to implement ZeroMQ is available on Github right now, in the branch “zeromq”. In an upcoming blog post I will post a developers update, to adjust existing plugins so they work with the new system (the changes are minimal, promised!)

Feedback on the new architecture and code is greatly appreciated!

After you have done your testing, please make sure you re-enable your firewall. As it is a good practice to always have it enabled. Here’s how to re-enable it:

Step 1: installing HouseAgent prerequisites

The first step is to install the HouseAgent prerequisites, there are two (Erlang and RabbitMQ) but you can install them using a single command line.
Start the terminal in Ubuntu:

Now enter the following command:

Hit “Y” to continue the installation.

Step 2: Installing HouseAgent

Download “HouseAgent.tar.gz” from the HouseAgent forum pages, and put it in a location of your liking. I chose HouseAgent in my home directory.
Now extract HouseAgent:

After extracting you need to copy the configuration file to /etc, you can do this with the following command line (you need to sudo this):

Step 3: Starting HouseAgent

You are now ready to start HouseAgent. You can do this by running: “./Houseagent” :

After HouseAgent has been started, point your browser to http://localhost:8080:

Congratulations, you have successfully installed HouseAgent! In tomorrow’s blog post we will look into installing the first plug-in in order to make HouseAgent actually useful.

Step 1: installation of RabbitMQ (and Erlang)

RabbitMQ is the message broker required by HouseAgent, you can download it from here.
You need the server with Windows installer:

After you have started the RabbitMQ setup, it will alert you that you also need Erlang:

Go ahead and download and install the latest Erlang version, choose the Windows binary form:

Now complete the installation of both Erlang and RabbitMQ, the default settings will suffice.

Step 2: Installation of HouseAgent

Download the latest beta release from the HouseAgent forum. And start the HouseAgent installation.
It’s an easy next->next->finish installation.

Step 3: Enjoy HouseAgent

Congratulations! You should now be ready to use HouseAgent!
There’s a HouseAgent start menu group in the start menu to get you going.

from plugins.pluginapi import PluginAPI

pluginapi = PluginAPI('922d6504-b58f-4d9a-b6eb-870aa06f2d00', 'TestPlugin')

The first argument is a unique GUID for the plug-in, a GUID can be created in the coordinator web interface:

The second argument is a plug-in type, this type can be used to identify plug-ins of the same kind. Optionally you can supply the following arguments:

• broker_ip (default: ’127.0.0.1′)
• broker_port (default: 5672)
• username (default: ‘guest’)
• password (default: ‘guest’)
• vhost (default: ‘/’)

These arguments are useful when you want to connect to a RabbitMQ broker other then on the localhost (yes, you could connect to a broker based in Sydney from Amsterdam!), or you could use different credentials for example. Of course you are not limited to setting up one plug-in connection, you could connect to multiple brokers for use in high availability scenario’s.
After you are connected to the broker you can use some built-in API commands to send data to the broker. Here is an value update example taken from one of the plug-ins I developed:

            values = {'Light': str(light), 'Humidity': str(humidity),
                      'Temperature': str(temperature), 'Motion': str(motion), 'Battery': str(battery)}

            pluginapi.value_update(node_id, values)

As you can see, you can supply an dictionary Python object with an unlimited number of values. What you do need to supply is the node address, this is an unique value to identify a device. A device is identified in the database based up-on a plug-in ID (which is the GUID) and a device address (which could be node number 1, 2, 3). The plug-in API also allows you to register to certain events, one example would be the power_on command (i.e. turn on a lamp module). Here is a basic registration, and callback function for the power_on command:

# Register ourselves to receive power-on callbacks
pluginapi.register_poweron(self)

# This is the callback function
def on_poweron(self, address):
    self._set_node_on(int(address))
    return {'processed': True}

All the helper functions supplied by the plug-in actually translate to AMQP messages, which can get quite complex. For the ease of development this is hidden from the developer. There is still an option though to talk directly to the broker (for example if you prefer a different language then Python). One other important thing to note is that error handling is done by the plug-in API, so you don’t need to take care about for example connection failures.

Plug-in GUI – Menu’s

HouseAgent allows you to create your own custom GUI for your plug-in, but you may still use standardized functions for your plug-in (for example for adding a device). You can also re-use the HouseAgent master template, for styling. One of the handy features is the menu.xml file, you can use this file to create your own custom menu items but still use the standard HouseAgent menu style which looks like this:

To create a custom menu you need to create a menu.xml file in the templates/plugins/yourpluginname/ folder. Here is an example of what the menu.xml file should look like:

<items>
	<item name="Latitude" url="#latitude" image="/images/plugins/latitude/latitude_icon.png">
		<subitem>
			<name>Latitude approval</name>
			<url>/latitude_approval</url>
			<description>Setup Google Latitude approval, to allow HouseAgent access to your location data.</description>
		</subitem>
		<subitem>
			<name>Manage accounts</name>
			<url>/latitude_accounts</url>
			<description>Manage Google Latitude accounts and approval.</description>
		</subitem>
		<subitem>
			<name>Manage known locations</name>
			<url>/latitude_locations</url>
			<description>Manage latitude known locations. You can for example set your home location in this page.</description>
		</subitem>
	</item>
</items>

The structure of this file speaks for itself.

Plug-in GUI – Pages
Now that we can create our own menu entry’s and get them visible in HouseAgent it’s time to actually create those pages. In order to host custom pages for your plug-in you need to create a file in the pages/ folder. You can just create a yourpluginname.py file inside that folder. There is one requirement, and that is to create an entry point function inside your pages file. The entry point function (exact name must be init_pages!) looks like this:

def init_pages(web, coordinator, db):
    web.putChild("zwave_add", Zwave_add(coordinator, db))
    web.putChild("zwave_networkinfo", Zwave_networkinfo(coordinator, db))
    web.putChild("zwave_added", Zwave_added(coordinator, db))

As you can see there are three important arguments passed, argument one (web) is an instance to the web controller of HouseAgent. You can extend this object to allow your own pages, this is done using web.putChild() function as listed above. The second argument is an instance of the coordinator, you can use this to interact with the broker (for example send a power_on command to the network). The last argument is an instance of the database, you can use this to interact with the HouseAgent database. Once you have defined your entry point function you can use web.putChild to extend the web object with a custom page class. Here’s an example of a custom web page (zwave_add):

class Zwave_add(Resource):
    """
    Class that shows an add form to add a z-wave device to the HouseAgent database.
    """
    def __init__(self, coordinator, db):
        Resource.__init__(self)
        self.coordinator = coordinator
        self.db = db

    def result(self, result):

        lookup = TemplateLookup(directories=['templates/'])
        template = Template(filename='templates/plugins/zwave/add.html', lookup=lookup)

        self.request.write(str(template.render(result=result[1], locations=result[0], node=self.node, pluginid=self.pluginid, pluginguid=self.pluginguid)))
        self.request.finish()

    def render_GET(self, request):

        self.request = request
        self.node = request.args["node"][0]
        self.pluginguid = request.args["pluginguid"][0]
        self.pluginid = request.args["pluginid"][0]

        deferlist = []
        deferlist.append(self.db.query_locations())
        deferlist.append(self.coordinator.send_custom(self.pluginguid, "get_nodevalues", {'node': self.node}))
        d = defer.gatherResults(deferlist)
        d.addCallback(self.result)

        return NOT_DONE_YET

This page is actually a bit complex, because there is some interaction with the coordinator. But it should give you an idea of what custom pages look like.

Plug-in GUI – Templates
HouseAgent also allows you to use templates to display data. For this the Mako template engine is used. As you can see in the above example page there is a template function which refers to “templates/plugins/zwave/add.html”. This is actually a Mako template file which looks like this:

<%inherit file="/master.html"/>
<%def name="content()">
<div id="output"></div>
<table cellspacing="0" cellpadding="0" align="center" width="100%" id="devices">
	<tbody>
		<tr>
			<td class="HeadText">Add z-wave device to the HouseAgent database</td>
		</tr>
		<tr>
			<td class="body info">
				<p>This page allows you to add this device to the database. Please select the values you would like to log..</p>
			</td>
		</tr>
        <tr>
            <td class="body">
                <form name="device_add">
                    <table class="InfoTable" cellspacing="0" cellpadding="2" border="0" width="40%">
                        <tbody>
                            <tr>
                                <td class="Head2" colspan="3">Device properties</td>
                            </tr>
                            <form>
                            <tr>
                                <td class="Label">
                                    <span class="requiredfield">*</span>
                                    Device name:
                                </td>
                                <td>
                                    <input class="Field250" name="name" id="name">
                                </td>
                            </tr>
                            <tr>
                                <td class="Label">
                                    <span class="requiredfield">*</span>
                                    Location:
                                </td>
                                <td>
                                    <select id="locations">
                                        % for location in locations:
                                            <option value="${location[0]}">${location[1]}</option>
                                        % endfor
                                    </select>
                                </td>
                            </tr>
                        </tbody>
                    </table>
                </form>
            </td>
        </tr>
        <tr>
            <td class="body">
                <table class="InfoTable" cellspacing="0" cellpadding="0" border="0" width="40%">
                    <tbody>
                        <tr class="HeadTable">
                            <td nowrap="" width="5%"></td>
                            <td nowrap="" width="30%">Command Class</td>
                            <td nowrap="" width="30%">Label</td>
                            <td nowrap="" width="30%">Value</td>
                        </tr>
                        % for value, details in result.iteritems():
                        <tr class="Row">
                            <td><input type="checkbox" id=${value}></td>
                            <td>${details["class"]}</td>
                            <td>${details["label"]}</td>
                            <td>${details["value"]} ${details["units"]}</td>
                        </tr>
                        % endfor
                    </tbody>
                </table>
                <br>
                <table class="PanelPlain" cellspacing="0" cellpadding="2" border="0" width="100%">
                    <tbody>
                        <tr>
                            <td height="30" valign="top">
                                <button id="adddevice_button">Add device</button><br><br>
                            </td>
                        </tr>
                    </tbody>
                </table>
            </td>
        </tr>
	</tbody>
</table>
</%def>

In the first line we extend from the master.html and re-use that style to create our plug-in page. One other important thing used here is for loop in the template to display locations in a select box. You can use all of the Mako templates features which is beyond the topic of this blog post to describe. But, I suggest you check out the Mako template website for more information.
This concludes the post about HouseAgent plug-ins. If anything is unclear please leave a comment or use the HouseAgent forum.

That was the short introduction of something I have been working on for quite some months already. HouseAgent has been running as my own home automation software for some time.
I hear you asking, why not use one of the standard software products available? There are a couple of reasons for that:

• Most software is expensive, for a full set-up €500,- is no exception;
• I wanted to learn from my home automation hobby, by writing my own software I got and still get a lot of insight on new frameworks/protocols/techniques etc;
• Really, I wanted free home automation software to be available to others out there as well!
• Most software out there is not designed to run on a low power/low profile systems. HouseAgent can be run on low power system without hassle.
• Also most software only runs on one platform i.e. Windows or Linux. I believe in a truly open platform where all kinds of developers can contribute.

Let’s look at some core components used in HouseAgent:

AMQP/RabbitMQ
One of the core components used in HouseAgent is AMQP. The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.
AMQP is used within HouseAgent for plugin communication. Here is a picture, which hopefully explains all the bits and pieces a bit better:

As you can see all plug-ins are separate isolated components, in fact the plug-ins don’t even know about the coordinator or each other. The coordinator is the main HouseAgent application which runs important features such as the event handler and the web-based GUI. A plug-in communicates with the broker, and the message protocol used for this is AMQP.
Certain types of messages are “published” on the broker, for example a device_value_update message might be send to the broker by a plug-in. The coordinator is “subscribed” to such messages and handles them appropriately (i.e. run a certain action based upon an event or save history to a history file)
The coordinator on it’s turn can publish messages to the broker as well, commanding the plug-ins to do certain action. One example would be the power_on message. This message commands a certain plug-in to turn on a device with a certain address.

So what’s RabbitMQ? RabbitMQ is a broker which implements AMQP, there are several other AMQP implementations which might work as well with HouseAgent (not tested!). You can read more about AMQP on it’s wikipedia page. You can read more about RabbitMQ on it’s homepage.

Twisted (Matrix)
Twisted is used for serial communication and all networking communication in HouseAgent. Twisted is a networking engine written in Python, supporting numerous protocols. It contains a web server, numerous chat clients, chat servers, mail servers, and more. The main difference between Twisted and other networking engine’s is that Twisted is asynchronous. You can read about the differences between synchronous and asynchronous communication here. Twisted certainly was an eye opener for (thanks Rene) but also required a completely different way of thinking about networking and communication in general.

Among these core components some other frameworks/packages are used:

• Mako Template (for web page templating)
• PyRRD (for historic graphing)
• py-serial (serial communication)
• pywin32 (Windows only)

Although the development of HouseAgent is just getting started, it already includes some nice features. Some core functionality HouseAgent offers right now:

• Storing and logging of device values
• Graphing (based upon RRDtool)
• Event handling (triggers include: time based, device value changes)
• Flexible plug-in development and easy extension

This concludes the short introduction of HouseAgent. In future blog posts I will dive deeper in different aspects of HouseAgent.
Right now a group of beta testers has been selected, I will send out a first beta version a.s.a.p. after that a public version will come available.
As of today the source code is available for the core of HouseAgent! You can review the source code, and watch the development progress on Github. But please don’t expect a complete system out that source code, stand-by for binary releases.

• Previously I mentioned an “ERASE” mode, this is actually “EXCLUDE” mode. This mode should only be used if your thermostat was previously installed in a z-wave network, I guess this was the case when I first set-up my thermostat. The protocol reset does more or less the same but is more extensive. It removes any network information from the HRT4-ZW and returns it to the factory default state (The double press of the dial is to avoid accidental operation)
• The way incorporate the devices with a controller (in my case the Aeon Labs Z-Stick) is to start with both in an unassigned state. Install each device separately on to your controller. The Association between the two devices may be done using the ‘A’ command on the HRT4-ZW and pressing the install button on the ASR. This does not require knowledge of the node numbers. The install (network) button is required for installation. You need to hold the button down for over a second.
• I forgot to mention about the wake-up time, I changed the wake-up time while it was in the LI (listen) mode. The default wake-up time is around 23 hours, which is a bit long when you want your thermostat changed. I changed it to 256 seconds or in other words 4 minutes and 16 seconds (the bare minimum value), but I might change it to a higher value again later on.. please see the point below why..
• The 2 year battery life (as reported by Horstmann) is based on normal use as a stand-alone thermostat. Use in a network where the HRT4-ZW is woken up regularly will reduce this time. That’s why there is a minimum wake-up interval. Battery life should not be a problem in practice, but please note that when the HRT4-ZW is in installer mode (switch 1 up) the Z-Wave module is permanently powered and takes much more power. This is why my battery life reported 90% after some tests. 0% is 2V and 100% is 3V or above. A new set of batteries will normally give just over 3V (2xAAA, 1.5v each)
• There is actually a second and third configuration parameter I forgot to mention about. The second one allows you to change to Fahrenheit messages. The third one allows you to set a delta t for reporting temperature changes. A message will be sent to associated nodes if the temperature changes by delta t. This parameter defaults to 1°C.
• When controlled by Z-Wave messages rather than turning the dial, you can actually set the setpoint to 0.1°C steps. This will not show up on the display as it’s only 2 digits, but it will attempt to control to it. For example you could set the setpoint to 20.6°C if you wished. 21°C will be shown on the display as the demand temperature is rounded up. Fahrenheit temperatures are converted internally to the nearest 0.1°C so will not give you more accuracy.

Some really valuable additional information, thanks a lot Mr.Anonymous ! :-)