tech-guide

Add Ethernet to your ChromeCast for 12 bucks

ChromeCast’s are great… as long as your wifi is. I upgraded my router firmware, and my ChromeCast started dropping out. Rather than fixing that problem, I figured “why not just hardwire the damn thing?”.

Google offer a ChromeCast ethernet adaptor, which is great except…

  1. It is never in stock
  2. It isn’t available in Australia

I’m sure it will become available in Australia sooner or later, but because we’re a country full of suckers surrounded by a huge fucking shark filled moat - it will probably sell for $40. I’m cheap and impatient, so here is my $12 DIY ChromeCast ethernet setup.

Things you will need

  • OTG adaptor with power feed
  • USB Network adaptor
  • A network point
  • Ethernet cable

OTG Adaptor

OTG means “USB-On-The-Go”, which means “you can plug normal USB things into micro-usb devices.” This means you can plug your USB drive into your Galaxy Edge, or your gaming mouse into your Motorola Moto-G, or ethernet into your ChromeCast.

There is a catch. Most OTG cables do not provide any way to get power, as they expect the device to provide power. This is no good for a ChromeCast, as a ChromeCast is USB powered. What you need is a “OTG Y-Cable”.

I used this one - $5.99.

USB Network Adaptor

The next thing you need is a USB Network adaptor.

I used this model - $6.85.

It works fine and it lights up blue to indicate network activity. It also works with a Macbook Pro, which is nice.

The nice folk on the Whirlpool forums have been discussing alternative network adaptors that may work - Whirlpool Thread.

Connecting it together

You won’t need your existing ChromeCast micro-USB cable any more. Connect everything else as follows:

  1. The OTG micro-USB plug should connect to the ChromeCast.
  2. Connect the Ethernet adaptor to the OTG USB female end.
  3. Connect the USB plug to either a powered USB port on your TV, or the ChromeCast power brick.
  4. Plug the ChromeCast HDMI plug into your TV.
  5. Plug your ethernet cable into the network adaptor.

how to connect stuff

Note that you still need to use WiFi to configure your ChromeCast initially. If your ChromeCast is already configured, it will work over ethernet without any changes.

Everything should now work.

activity indicator Look at the neat activity indicator!

Verifying

For the tech minded readers, if you jump on your router configuration page you should notice a new device. The ChromeCast will have two IP addresses allocated, one for wifi, and one for ethernet.

If you fire up NMap, and run a port scan, you will get something similar to this:


$ sudo nmap -sV 192.168.1.72

Starting Nmap 6.49BETA3 ( https://nmap.org ) at 2015-08-31 20:29 AEST
Nmap scan report for 192.168.1.72
Host is up (0.0030s latency).
Not shown: 998 closed ports

PORT     STATE SERVICE    VERSION
8008/tcp open  http       Google ChromeCast httpd
8009/tcp open  ssl/castv2 Ninja Sphere ChromeCast driver

MAC Address: 00:E0:4C:36:00:00 (Realtek Semiconductor)
Service Info: Device: media device

Nice.

If you’re Australian, and running a VPN or DNS redirect (for delicious international netflix), don’t forget to block the Google DNS servers (8.8.8.8 and 8.8.4.4) for your ChromeCast IP address.

Time to fire up Netflix and relax.

/eof

How to Upgrade Windows 8 to Windows 10

The Windows 10 consumer preview is here! If you’re already running Windows 8, the upgrade is simple. If you have a spare couple of hours and would like to run a slightly-buggy pre-release operating system - you’re in luck. Here is how you do it.

The Windows 10 Upgrade Process

The installer can be downloaded here: Windows 10 Preview.

After running the installer file, this is how the install will progress:

  1. Installer splash screen, click the “Start Upgrade” button.
  2. Windows update runs and downloads a couple of GB of updates. Press the “Get Started” button.
  3. Eula, click the “I accept” button.
  4. Windows update will load, click the “Restart now” button.
  5. Your current version of Windows boots back up. Click the “Start the upgrade now” button.
  6. Finally, we boot into the OS installer. Expect to be here for one hour plus, even if you have an SSD.
  7. Windows 10 boots and asks basic setup questions. Note that the “Express Setup” option will send your data (web browsing, crash reports etc) to Microsoft. If that worries you, choose “Customise”.
  8. After answering the questions you will finally be able to log in. Hooray!
  9. Windows 10 will spend 5-10 minutes building caches, installing drivers, and doing other under-the-hood things. My laptop went from low resolution back to normal native res after about 5 minutes. Before attempting any serious work, reboot! I experienced out-of-memory errors and other oddities until I rebooted.

Welcome to Windows 10!

Windows 10 setup screen

Errors / Items Encountered

Customise vs Express

If you choose “Customise” instead of express setup, one of the questions asked is “can we send your browsing history/requests to Microsoft to allow pages to be preloaded for faster browsing”. So, all your odd fetish porn habits will be sent to Microsoft, but on the upside your disturbing turn-ons will pre-load for you. Good news?

Ignore the Percentage Bar

When you reboot to the Windows 10 installer, the percentage bar doesn’t operate on normal time. I spent a long time at 39%.

Memory Errors

On first boot, Spartan was broken (wouldn’t load anything), and I got “out of memory” warnings. A reboot fixed all that.

Stuck in Low Resolution

Windows 10 initially displayed in glorious 1024x768. Wait for 5 - 10 minutes, Windows should find and load the right driver. Note that Windows 88.1 drivers should work in Windows 10.

Windows 8 Start Screen Items

If you are a Windows 88.1 user, everything you pinned to your start screen is gone (the shortcuts are gone, not the apps). You can still find your short cuts using the Search function on the start menu. If you right click the search result item, you can pin it to your new start menu.

Setup Screen Shots

Here are some photos from the upgrade process.

Click the button to download the installer.

*Windows Update doing its thing.*

You're going to spend a lot of time here. At least 1 hour, probably more

/eof

Chromecast issues? Check your Multicast settings

I’ve had network gremlins for a few months now. Sometimes my WeMo switches are visible, sometimes not. Chromecast is the same. My devices were screaming for attention, but my router wasn’t listening. Lets look at Multicasting (and Broadcasting).

What is broadcasting?

BroadcastAt a basic level, a network broadcast is when a device spams the whole network with a message.

In computer networking, broadcasting refers to transmitting a packet that will be received by every device on the network.

Wikipedia

Why would a device want to do this? Well, lots of reasons.

Take your home network connection for example. When your laptop connects to your home network, it broadcasts a DHCP request. Your router replies with an ipaddress and off you go.

You should also be aware of Multicast.

What is multicasting?

MulticastMulticast is similar to broadcast - we are sending a message to multiple nodes, but in this case, we are sending it to a subset of nodes (rather than spamming everyone).

As far as your router is concerned, broadcasts and multicasts are very similar. If you have an issue with one, you probably have an issue with the other.


The symptoms of broken Multicasting

In my case, I had two issues that indicated multicasts were not configured properly.

The first issue was my Belkin WeMo light switch and power socket. These are network aware IP devices that can be switched on or off with my phone. I could control them via 3g, but not over my LAN. This seemed odd, but makes sense when you learn how the WeMo devices work.

Broadcast
Discovery Protocol in action

When away from your home network, WeMo devices register through Belkin’s cloud service. They then accept push messages. So, your phone sends a “light on” command to the WeMo cloud, the WeMo cloud then sends a push message to the switch.

On your home network, the cloud is not used - uPnP kicks in. Your phone sends a multicast discovery message along the lines of “HEY! any WeMo devices out there?”. The WeMo devices pick that up, kick off a handshake with your phone, and away you go.

If the initial discovery multicast message is not received, the WeMo devices are not found.

My second issue was with my ChromeCast. Sometimes it would be discovered by my phone, sometimes not. ChromeCast works by using the DIscovery And Launch (DIAL) protocol. The discovery part uses Simple Service Discovery Protocol (SSDP), sending a multicast message to the address 239.255.255.250. The ChromeCast response is a uPnP UDP message, containing address details for the ChromeCast device.

If the initial Multicast packet is not being sent/received, the ChromeCast will never respond - meaning my phone will never find it.

My router had some multicasting options configured poorly, however uPnP was turned on. This resulted in intermittent discovery of the ChromeCast.

As an interesting side note, the DIAL protocol was co-developed by Netflix and YouTube (with help from Sony and Samsung). My buddy Jase did something similar with discoverable sound hardware a few years back - but he didn’t give it a cool name, so no one heard about it.

Suggested Router Settings

The settings below are the suggested settings for working multicasting on your home WiFi network.

AP Isolation

Possible settings

  • on / off

Description: If turned on, this will prevent wireless clients from messaging each other directly. This means no fileshares, no network games, no printing, no nothing. All you can talk to is the router. Unless you are an Internet Cafe, this should be off.

IGMP Snooping

Possible settings

  • on / off

Description: As per Wikipedia: is the process of listening to Internet Group Management Protocol (IGMP) network traffic. The feature allows a network switch to listen in on the IGMP conversation between hosts and routers. By listening to these conversations the switch maintains a map of which links need which IP multicast streams. Multicasts may be filtered from the links which do not need them and thus controls which ports receive specific multicast traffic.

Many multicast capable media devices are known to need this setting turned on. This includes the AppleTV, and ChromeCast.

Multicast Rate

Possible settings

  • Auto
  • HTMIX - (speed)
  • OFDM - (speed)
  • CCK - (speed)

Suggested setting

  • HTMIX 130144

Description:

The first part refers to the modulation, the second part the speed. Each modulation is (generally) aligned with a Wireless speed/standard:

  • CCK - 802.11b
  • OFDM - 802.11g
  • CCK - 802.11b

The speed portion is (min Mbit/s)/(max MBit/sec). The general rule is that the minimum should be a little higher than the maximum bandwidth your streaming device will use. Note that the higher setting you use here, the more you will restrict your WiFi range.

Beacon Interval

Possible settings

  • Beacon Interval: (number)

Suggested setting

  • 50ms

Description:

Your router periodically sends a “Beacon” packet. This does several things:

  1. Includes a Date/Time used by devices to syncronise
  2. Advertises the network capabilities
  3. Announce the SSD and presence of the WiFi router.
  4. Acts as a “heartbeat” packet

Generally, decreasing the beacon setting will mean more administrative overhead for your network, but you gain increased reliability. Some devices (Apple TV again) require 50ms or less to work.

Summary

After the changes have been made above, you will probably need to restart your router. After I made the above changes, I found all WeMo devices were visible and the ChromeCast icon was now appearing in NetFlix. Happy days!

References

The multicast and broadcast images on this page are sourced from Wikipedia.

/eof

How I Got Burned Today - Windows 8 Install

Windows 8 polarises people. I’ve never minded it. It performs well in spite of having a split personality. Shiny new SSD in hand, I attempted a clean install on my laptop. Windows 8 pushed me from my usual slightly agitated states towards a murderous rage.

The Install Plan

My Crucial C500 SSD arrived from Amazon yesterday (coincidently a new improved model was released today), so I planned a fresh OS install on my laptop. My long suffering laptop came complete with a metric ton of Hewlett Packard crapware, which claims to keep my system running smoothly. In the spirit of the HP mothership - it overreaches, and falls flat on its face.

My laptop came with Windows 7, which I upgraded to Windows 8 at some point, so I had a combination of Win7/Win8 along with some lost and confused HP “software”. The HP “software” seemed to team up with Skype and push my CPU utilisation to 100%, causing the cooling fan to scream like a jet engine for most of the day.

My re-install plan was simple:

  • use the Windows 8 ISO to make a bootable USB installer
  • remove old hard drive and place in USB3 enclosure
  • install SSD
  • install Windows 8 on SSD from USB stick
  • restore data from USB3 enclosure

Windows 8 had other ideas.

Trying to Partition Your Drive? No.

At about 9pm last night I was at the point of installing Windows 8. The installer booted up, and things seemed to be going well.

Then it came time to partition the drive. I created the partitions, then clicked next.

An ominous message appeared:

We couldn't create a new partition or locate an existing one.

I figured I’d done something wrong, so I restarted the whole process (twice!) only to be greated with the same message. No luck.

The next step was to try creating the partitions manually. It didn’t work, but here is the process:



// Boot up the command prompt (boot off DVD/USB, press shift-f10 when you get past the welcome screen) run these commands:

DISKPART
LIST DISK   (identify your SSD disk number (from 0 to n disks))
SELECT DISK <n>  (where <n> is your SSD disk number)
CLEAN
CREATE PARTITION PRIMARY
ACTIVE
FORMAT FS=NTFS QUICK
ASSIGN

// now reboot

I fired up the installer again.

We couldn't create a new partition or locate an existing one.

Argh!

toot toot

Windows 8 won’t install off of a USB stick

I did a bit of research and discovered that Windows 8 won’t install off a USB stick!

Is this really such an uncommon scenario? Windows 7 will install off of anything. I could whack a DVD against the screen of my laptop and Windows 7 will magically end up installed. Even geriatric old Windows XP would happily install from USB.

What on earth have Microsoft done to break the Windows 8 installer so badly?

Windows 8.1 appears to be fixed, but that’s no help because a Windows 8 key is not valid to install Windows 8.1. On what planet did Microsoft think this would be a good idea?

I suspect there is probably some obscure technical reason, but wow.

So, what is the solution?

You can try creating the partitions manually, as I tried above. It didn’t work for me, but apparently did work for some smug people on the internet. What did work for me, was copying the installation files to the SSD.

Yes, you read that correctly. You have to copy the installation files onto the SSD… then the SSD can install Windows 8 on itself. Winception?

You're not in Expansys anymore, Toto.

To do this, start the repair option in the installer, and head back to the command prompt. Then do this:


//Boot up the command prompt (boot off DVD/USB, press shift-f10 when you get past the welcome screen) and do this:

// use DIR commands to figure out which drive is your USB drive, and which is your SSD
DIR /A C:
DIR /A D:
DIR /A E:

// copy USB drive files to your SSD
// in my case the command was XCOPY /E /H /R E:\ C:\
XCOPY /E /H /R <source> <target>

// now take the USB drive out and reboot. You should be in a working window 8 setup!

Problem solved, Windows 8 installed. Here is a page with lots of solutions (including the above) for problems you may face installing Windows from a USB stick. Check them out.

Now, time to activate.

The Windows Activation Process Hates You

I tried personalising a few things in Windows 8. Turns out you can’t until you have activated. No worries, I have my own legit Windows 8 key that I paid for. Windows 8 disagrees:

Windows cannot be activated.

Turns out you can’t do a fresh install from a Windows 8 upgrade edition. Of course, the Windows 8 installer doesn’t share this nugget when you type in your licence key.

chuga chugga derp derp

The Microsoft Website cheerfully recommends blowing away your fresh Windows 8 install, installing Windows 7, then upgrading to Windows 8.

If you formatted or replaced your hard drive, you won't be able to use a product key to update to Windows 8.1 or Windows 8. You'll need to install your previous version of Windows, and then reinstall Windows 8.1 or Windows 8.

Some Microsoft employee who hates you

Back in Windows 7 or earlier, an upgrade edition was fine, you just had to point the installer in the direction of your Windows XP DVD/folder/whatever to prove you owned it.

At 1am, that seemed about as fun as setting myself on fire, so I searched for another solution.

This kind person found a solution:

  1. Open regedit by pressing Windows-q, entering regedit and selecting the result from the list of hits.
  2. Navigate to HKEY_LOCAL_MACHINE/Software /Microsoft/Windows/CurrentVersion/Setup/OOBE/
  3. Change MediaBootInstall from 1 to 0
  4. Go back to the start screen and enter cmd there.
  5. Right-click Command Prompt and select to run it as administrator.
  6. Type slmgr /rearm on the command line and hit enter.
  7. Reboot Windows now.
  8. Run the activation utility afterwards, enter your product key to activate Windows.

As soon as I followed those instructions, I could activate!

Summary

Microsoft, fix your installer. The installer may be the first experience a user has with your product, especially the upgrade edition. It is the first step in building a positive relationship with your user.

The areas that need improvement are

  1. Your installer needs to work with a USB stick plugged in.
  2. Windows 8 and 8.1 keys should be cross compatible with each other.
  3. Even if I own the upgrade edition, there needs to be a method of doing a clean install. Sure, you may need a validation step (enter your old product key, or insert old boot media for example), but make it straight forward. Keep in mind that many users do not get installation media when they buy a pc. When they do, it is often loaded up with crap-ware. A fresh install will give the best possible experience.
  4. Please have a look at your product activation. I shouldn’t have to edit a registry entry to activate a product I have legally purchased.
  5. If I have a Windows 8 licence, let me download a Windows 8.1 iso with all the latest security patches. This would have saved me hours.

If installing and activating is too difficult, many users will run a pirate copy instead - or worse, defect to a competitor. My coworker Json is Windows guy from way back (he even remembers all the old Win32 system calls). He switched to a MacBook Pro. Since changing, the only issue he has is that the machine refuses to boot if it feels he isn’t smug enough. Json was making fun of my installation woes yesterday. You don’t give Json any more ammunition, do you? (no, you don’t).

It isn’t all bad news. The installer (when it works), looks slick. It is fast. Under the hood Windows 8.1 is great. I like the OS. Your service is improving too. I whinged about the Win8 installer on Twitter, and quickly received a response from ^Helen - who seems wonderful. Please give her a pay rise.

chuga chugga derp derp

/eof

Chocolatey - Package Management for Windows

There’s finally a package manager for Windows in the same style as Apt-Get, and it is surprisingly good. Lets get Chocolatey.

The Original Package Manager - Apt-Get

If you’ve spoken to a Debian user for more than 30 seconds, you’ve no doubt heard of apt-get - the all singing, all dancing, Debian package manager. If you need to install something, Apt will:

  • check dependencies, download, configure, and install/update them
  • install your application
  • place all the app components in standard locations within your file system.

Cool, huh? It gets cooler. Apt will even upgrade your entire operating system via the dist-upgrade command. Incredible.

There are five reasons apt works so well:

  1. Packages must adhere to strict rules about where libraries/config/logs are placed. If they fail to do so, the package is considered to be broken.
  2. Repositories are actively maintained.
  3. Dependencies are automatically managed.
  4. The package repositories contain pretty much anything you can think of.
  5. Apt-get is reliable.

Knowing what Apt can do, does Chocolatey measure up?

Just What is Chocolatey?

Chocolatey describes itself as follows:

Chocolatey is a global PowerShell execution engine using the NuGet packaging infrastructure. Think of it as the ultimate automation tool for Windows. think about Chocolatey as a global silent installer for applications and tools. It can also do configuration tasks and anything that you can do with PowerShell.

Nice.

Chocolatey has 7,636 packages. It is nowhere near the number available in apt-get repositories, but it is growing fast. Like apt, it manages your dependencies, and installs everything in a standard location (c:\Chocolatey). A root c:\ install location may seem bad at first glance, but it makes sense give that it manages packages per machine (not per user), and who wants to navigate to c:\Documents and Settings\All Users\etc\etc ?

Unlike Debian, Chocolatey has no control of the OS - so it is always going to be limited in terms of how strict it can be with the file system.

The way repositories work in Chocolatey is a little different too. Chocolatey doesn’t actually maintain binary repositories. It maintains the package files (which includes version information and install/configuration scripts), but the actual binary installer still comes from the “point-of-truth” for that app. This means that winzip will be downloaded from winzip.com and installed. The upside is that you are always getting the “official” version. The downside is that Chocolatey will never be as reliable as apt-get.

Using Chocolatey

Chocolatey in action

Chocolatey is pretty straight forward to use. The hardest part is typing the silly name. Here are the basic use cases…

Installing and Uninstalling

Installing a package is simple:

cinst <package name>

You do get prompted during the install to click “ok” due to Windows UAC security. The rest of the install is automatic.

You can install a specific version:

 chocolatey install <package name> -version x.x.x.x

To remove a package:

chocolatey uninstall <package name>

Searching

List all packages:

chocolatey list

Find a package:

chocolatey list <package name>

or:

chocolatey search <string>

Other interesting things about Chocolatey

  • Chocolatey has auto-packages, which are automatically created when you check a new version of your app into source control (after the initial set up of course).

  • Chocolatey can install Ruby Gems, and Python Packages

  • Chocolatey has “meta packages” that have no content, but exist solely for the purpose of keeping packages grouped and organised for dependency management

  • Chocolaty uses the Visual Studio package management tool (NuGet) & associated infrastructure

Summary

I’m now a huge fan of Chocolatey. It seems reliable, and makes it very quick and easy to find software. It looks to be a huge time saver for automating environment set ups.

This is applicable to both the work place, as well as open source projects. No longer will we need huge wiki pages of instructions to get a new Developer up to speed - just fire off a batch or powershell script with all your Chocolaty calls in it.

Give it a try today: Chocolatey

/eof

OCR Solar Meter Reader - Part 1

I had some solar panels installed last year. I wanted to read my inverter statistics, but I also wanted to log exactly how much power I import/export. Challenge - I can’t physically plug anything into the meter (power company regulations). What to do?

How about a web cam, raspberry pi, and OCR?

The Goal

The goal is to get all my data to log auto-magically to the PVOutput portal (pictured). Here is my PVOutput page.

Physical Layout

The Problem

I have an array of PV panels on my roof, which are connected to an inverter. The inverter feeds generated power back into the power grid via my power meter (the power company pays me for this).

I decided I would use a Raspberry Pi to read data directly from the solar inverter. That gives me stats such as the amount of power produced by the array. It doesn’t give me how much I actually export to the power grid - as that figure comes from the power meter.

The formula for exported power is:

Exported Power = Generated Power - Power Used by my house

So, how do I interface with my EM1000 power meter? The meter has a couple of ports:

  1. RS232 Serial Link
  2. Optical Port

Should be easy right?

Not quite.

The power company seal the RS232 port. Breaking those seals is a criminal offence. I did try to get permission to do it, but my provider (Energex) kept saying no. That leaves the Optical Port. Optical Readers are expensive, but a bigger problem is that Energex doesn’t use a standard protocol. They came up with their own (figure that one out, existing protocols just aren’t good enough read a number from a meter. I wonder how many hallucinogenic mushrooms were consumed to reach that decision).

So, what to do? The only other place data was visible was the LCD. Surely we can’t just take photos and OCR (Optical Character Recognition) the result? It can’t be that simple? Well…. kind of.

Physical Diagram

Let’s skip ahead to the physical diagram, and then look at how I got there. This is the physical set up of the system:

Physical Layout

Physical Setup

Raspberry PiA power box is fairly small, so I expected a challenge fitting everything in.

Getting the Pi in there was fairly easy. A piece of velcro tape keeps it firmly in place.

The Pi’s USB port does not provide enough juice to power anything more than a keyboard, so I have added a powered USB hub. That lets us power a light and a webcam.

For the webcam, I am using a Logitech C270. Due to the short distance between the cam and the display, I had trouble getting the camera to focus. For my first attempt, I mounted the camera above the power meter and used a mirror mounted on the door to get enough focus distance. This did work, but wasn’t great, so I took apart the camera and manually refocused it. Here is a guide on how to refocus the camera: YouTube: Refocus Logitech Webcam. It is easy, but you can kiss your warranty goodbye.

With the focus fixed, I was able to mount the camera properly. I used some cheap metal strapping from Bunnings, which I bent into an “L” shape. This was screwed into the top of the meter box. To secure the camera, I removed the camera clip and then screwed the camera directly into my home made bracket.

Lighting

Lighting was a particular challenge. For the display to be readable via OCR, there needs to be a consistent brightness across the entire display. There also needs to be enough contrast to clearly differentiate the characters against the background.

Fail LightI tried a flexible stick on strip light - but there were too many light sources, and they were too bright. I then tried an el-cheapo goose-neck USB light in a “fake lightbulb” housing - but that was too inconsistant. One side was dark, the other light!

Lighting up the boxI settled on a USB goose-neck reading light (found by accident in Myer). I mounted it higher than the LED display to minimise reflections. I was nearly there, but not quite, I was still getting too much glare. My first eureka moment was spraying the inside of the power box lid with matt black paint. That cut down the glare, but I still had contrast issues. My second eureka moment was moving the camera down, so it looks up at the display on a 15 degree angle. Suddenly I had a razor sharp well lit image! nice!

Network

I tried WiFi first, but found that putting a wifi adaptor in a big metal box makes reception near impossible (not too surprising, I guess). I didn’t want to run Cat6 cable to my power box (too much 240v power in the ceiling/wall cavities around there).

As luck would have it, I had some spare Netcomm Ethernet-Over-Power EoP adaptors sitting around. I plugged one in and away we went. If you haven’t seen these before, they are really neat. They use your existing electrical wiring and give speeds anywhere up to 500Mbit (half duplex!). Like everything, they have a catch. If your electrical wiring is bad, or you try to run them across circuits - performance will be abysmal. Certain appliances can also interfere with them. Have you ever tried troubleshooting a network where turning on a kettle (jug) kills connectivity?

Part 2

The next part details the software side of the project.

/eof

OCR Solar Meter Reader - Part 2

Continuation of Raspberry Pi Solar Data Logger article. In this part I detail the software side of the OCR solar data logger. Perl, APIs, C. Hooray!

The Software Part

With the hardware part sorted, now comes the easy part - software.

Getting inverter data is easy enough, but reading the meter is more of a challenge.

There are 3 basic parts to this:

  1. Take photos
  2. OCR the photos and store the values
  3. Send it to the PVOutput.com API

We also need something to tie it all together. I’d usually use my beloved DotNet stack, but this is Linux so that means Mono… and getting Mono working on a Raspberry Pi requires some form of animal sacrifice… or a prayer to Zeus… or something.

I considered using shell script, but then I remembered how much I hate shell scripting. I fired up my editor and hacked up some Perl. Why Perl? Perl is tried and tested, has good library support, and is available pretty much everywhere. Also, this is valid perl code which amuses me far more than it should:


    print'````'^RPQT

How can you not love a language where that is valid code? source. The only language sillier than that is BrainF*ck… god help anyone trying to do something productive with that.

Process Overview

Here is an flow chart of the process.

Physical Layout

Essentially, the pseudo-code version goes like this:

At 11:45pm take 20 photos
For each photo
    OCR photo
    If (value -ve) store export
    Else store import
If we don't have import/export explode, and notify admin
Store import/export in CSV
Calculate daily import/export by subtracting previous days values
Send daily import/export to PVI Output REST API

Taking Photos

I found a handy little command line tool for taking snaps - uvccapture. All it needs is a camera location and resolution and away it goes.

This is a photo taken with uvccapture:

Web cam photo

OCR time!

Figuring out how to OCR the display proved a bit tougher than I’d hoped. I found this great little app -Seven Segment Optical Character Recognition, SSOCR for short.

SSOCR was developed with the intent of reading RSA security tokens, allowing one token to be easily shared by a group of people - which completely circumvents the point of having a RSA token.

The gentleman who wrote it is a stern German who appears to still live in 2005, in a world devoid of joy, happiness, and code sharing tools such as GitHub. He’s aware of GitHub, but refuses to use it. I’m sure he’s a very nice guy, but he needs to be slapped into 2014.

Moving along.

SSOCR is written in C, and has the source code available - allowing us to modify it to work with something other than an ancient RSA token.

The changes I had to make are:

  • option to ignore decimal points
  • option to allow minus signs (for negative numbers). Only the first character is checked.
  • option to allow blank characters
  • recognise a 9 with the lower segment not set

I emailed the stern German with a list of the changes I made, but neglected to email my patch through (I did intend to!). In efficient German fashion he has implemented all of those things on his own (they are all in the latest version of SSOCR).

SSOCR is really cool. It scans the image, line by line, and attempts to identify line segments. It then uses the segment data to figure out which number was discovered.

Here is some debug output from SSOCR:

OCR Output

The different coloured lines show where SSOCR has identified line segments. Neat.

Once I have the SSOCR output, my perl script store it in a CSV file.

Hello, PVOutput API

PVOutput provide a handy REST API for pushing your data in. As much as REST is straightforward enough, why can’t the damn thing provide a WSDL like SOAP? instead, I have to type out all my requests like some sort of neandrathal. On the plus side, I don’t have to deal with 3 miles of XML gibberish, so there’s that.

This is how easy it is to submit data:


    my $ua = LWP::UserAgent->new;
    $ua->default_header(
                    "X-Pvoutput-Apikey" => $pvoutput_apikey,
                    "X-Pvoutput-SystemId" => $pvoutput_sysid,
                    "Content-Type" => "application/x-www-form-urlencoded");

    my $pvoutput_url = "http://pvoutput.org/service/r2/addoutput.jsp";
    my $request = POST $pvoutput_url, [ d => $date, e => $export*1000), ip => ($import*1000) ];

    my $res = $ua->request($request);

    if (! $res->is_success) {
        die "Couldn't submit data to pvoutput.org:" . $res->status_line . "\n";
    }

There is another reason to love perl. If things go wrong, we don’t exit, or error like lesser languages. No, we commit seppiku. “I can’t fulfil this task you ask of me, so I’m going to die”.

Perl is great.

Getting Inverter Data

I use Curtronics Aurora program to retrieve statistics from the inverter, once per minute. These statistics are logged and stored in daily CSV files.

Data is sent to PVOutput every two minutes, using the pvoutput-integration-service.

Once running, this service is set and forget. Just remember to add a start up script for it, so it comes back up if the Pi goes down for any reason.

So, between the pvoutput integration service - and my OCR magic above, both the inverter data and meter data are being sent to the pvoutput portal. Hooray!

When it all goes wrong

Did you know the Raspberry Pi has a hardware Watchdog Timer? Overview:

A watchdog timer is a special kind of timer commonly found on embedded systems that is used to detect when the running software is hung up on some task. The watchdog timer is basically a countdown timer that counts from some initial value down to zero. When zero is reached, the watchdog timer understands that the system is hung up and resets it.

The quote above is from Ricardo’s Workbench. He explains how to run it on your Pi. So, now my Pi reboots itself if it hangs.

Very nice.

Conclusion

It has taken a long time, and lots of experimenting to get the solar meter reader working reliably. I’ve had a lot of fun doing it, and learned a few things.

I went up the garden path a few times. For example: when I was having trouble getting the OCR to work reliably, I wrote a Node.js based site for selecting photos from the meter and updating pvoutput. That’s a future blog post.

I’m really blown away by how far things have come in the last few years. x86 PCs the size of a mouse? Incredible. I can’t wait to see what the next few years bring.

/eof

Automatically post your blog to social media

A good friend of mine recently said “it’s a pity I can’t just automatically post my blog entries on social media”. Good news! You can - with one caveat… you can everywhere except Google+.

If-That-Then-This (IFTTT) is a fantastic (free!) integration tool. I’ll show you how to leverage it.

First, a note on google. How can a company founded, staffed, and loved by developers be so terrible at APIs? A good public API allows developers to integrate. We make tools, we automate things, we develop apps that automatically post a picture of a cat on twitter when your WiFi aware light switch is turned on or off (true story).

So, if any Googlers read this - please, please open your APIs!

Automatically post to Twitter using IFTTT - Step By Step

Go to IFTTT and sign up. Before starting, you are going to need to activate the Twitter and Feed channels. When you activate a channel, you need to configure all the associated settings for the service. In the case of twitter, you need to sign in your twitter account.

IFTTT organises all of your tasks into recipes. We are going to create a new recipe.

  1. The IF part of If-This-Then-That refers to a trigger condition. In our case, it is a new blog post appearing in our blog’s ATOM feed. Choose the “Feed” channel. Feed Channel

  2. Choose “New Feed Item” as the trigger. Choosing the trigger

  3. Enter your blog Feed URL. Trigger Details

  4. Set “Twitter” - “Post new Tweet” as the action. This is the That part of If-This-Then-That. Set an action

  5. Add some descriptive text. You can reference the title and URL using variables, as pictured. Add the title and URL

  6. Confirm your new recipe, and you are good to go. Add a new blog post as a test. Confirm it

You can add a recipe for Facebook following the exact same process. IFTTT really is a great service. Here are some other things I do with it:

  • Sending a push message to my phone when my porch light is turned on
  • Sending a push message to my phone when my irrigation pump is turned on or off
  • Sending an email to me if it is going to rail tomorrow
  • Turning my porch light off at Sunrise.

Happy automating!

/eof