Category Archives: Meetups/Conventions

IoT, Learning, Meetups/Conventions, NodeJS, REST

My ODTUG GeekAThon 2017 Entry

September 26, 2017 bcarter Leave a comment

The rules and other information can be found at ODTUG GeekAThon 2017.

Problem

My son Alex attends a school where the students have some ‘bonus features’, or as the school puts it: “Educating Exceptional People”. There are some students at Alex’s school who sometimes try to wander away. Obviously, this could be a problem but the school staff is extremely well trained and they keep a close watchful eye on all of the students. Still, I’d like to try and make their lives a little easier, and the students a little safer.

There are commercial systems available that could notify the administration and/or lock doors when a beacon worn by a student is detected in a hazardous zone, such as leaving the school. That sounds perfect. There’s just one problem: those systems can be very expensive.

Proposal

Implement a student tracking and door lock automation system that can operate on inexpensive components and open source the software. I will set up a test environment at my house and my son will test it with me.

Desired Features

Central to the whole system is a way to detect a beacon when it enters a specific area such as near an exit door or a faculty-only area.
Ability to send notifications.
Ability to trigger a physical event such as a door lock or audible alert.
Log beacon detection events in a database.
- Beacon Id.
- Distance from the scanner.
- Timestamp.
Affordable components.

Initial Idea

After browsing the web for a while I decided I would set up multiple scanners with overlapping zones then use trilateration (I like saying that word) to determine the position of the beacon.

I would set up multiple scanners, measure the distances between them and plug that data into my database. When a scanner detects a beacon it would use my ORDS service to POST its own id, the beacon id and the calculated distance to the beacon. On the database, I would use Oracle Spatial queries to determine the location of the beacon. Finally, I would compare the beacon location to defined zones in my house and trigger the alerts/actions for the zones.

I have a tendency to over-engineer my projects. I once built a doghouse that weighed close to 200 lbs. (It was awesome.)

After getting most of this working, I realized that I could achieve the project goals by simply placing a single scanner near each zone and let that scanner initiate the alert actions for its zone. Sometimes less is more.

Hardware

I already had a bunch of Raspberry Pi so I decided to use a couple of my Pi 3s. Since I’m always looking for an excuse to buy more toys, I decided to get a Pi Zero W.

I have a z-wave enabled deadbolt and a Z-Stick USB hub that I can control using Home-Assistant.io. For the audio notification, I’ll push a ‘text to speech’ action to my Sonos speaker. I can make the Sonos say anything I want, this entertains me a lot, my family… not so much.

Software

Raspbian Linux
- Linux modules
  - bluetooth
  - bluez
  - libbluetooth-dev
  - libudev-dev
NodeJS
- NodeJS Modules
  - bleacon
  - request
IFTTT.com
Home-Assistant.io
Oracle Database
Oracle Rest Data Services (ORDS)

The installation instructions are in the GitHub repo.

Database

The beacons are set to transmit every two seconds and can be detected by multiple sensors. I always like to keep track of my data so of course, I’m pushing it to a database. I’m using an Oracle Cloud Database with an ORDS (Oracle Rest Data Services) front end to collect the data. When a Raspberry Pi detects a beacon, it will calculate the distance then POST the data to the database. The database will automatically record a time-stamp when the record is inserted.

{"scannerId": scannerID, //The ide for this scanner
"beaconId": beaconId,    //The detected beacon's ID
"distance": distance}    //Calculated distance between the beacon and scanner

{"scannerId": scannerID, //The ide for this scanner

"beaconId": beaconId, //The detected beacon's ID

"distance": distance} //Calculated distance between the beacon and scanner

This is included in the current code and it’s what I need to collect the data for the “Initial Idea” section above.

If I decide to implement the feature to track the beacon’s position throughout my house. I just need to determine the fixed position of each scanner relative to a point in my house and using the data I’m already collecting, run an Oracle Spatial query that defines a circle from each scanner with a radius of the distance to the beacon. Where the circles overlap is the approximate location of the beacon. The official term (linked above) is Trilateration, but you can think of it as a Venn Diagram.

How I Deployed the System

If you’d rather, you can watch the video and skip this section.

I configured and positioned three Raspberry Pi through my house. I put a Pi 3 in the hallway outside of the bedrooms, a Pi 3 near the front door and a Pi Zero W outside on the front porch.

The first Raspberry Pi 3 in the hallway is set to trigger an alert when the beacon is approximately 2 meters away. This alert will send a notification through IFTTT* to the app on my phone.
The Raspberry Pi 3 near the front door fires an alert when the beacon is approximately 1 meter away.
This alert has three actions:
- Send the ‘lock’ command to the deadbolt through the REST interface of Home-Assistant.io using Z-Wave. Home-Assistant.io and the Z-Wave USB dongle are also installed on this Pi.
- Set the Sonos volume to max and send ‘Locking the front door’ to the Sonos speaker using the text to speech function in Home-Assistant.io.
- Send a notification through IFTTT to my cell phone.
The Raspberry Pi Zero W outside near the front door will trigger an alert when the beacon is approximately 1 meter away.
This alert has three actions:
- Send the ‘unlock’ command to the deadbolt through the REST interface of Home-Assistant.io using Z-Wave.
  (If Alex makes it outside, I want the door unlocked so he can come back in.)
- Set the Sonos volume to max and send ‘Unlocking the front door’ to the Sonos speaker using the text to speech function in Home-Assistant.io.
- Send a notification through IFTTT to my cell phone.

*IFTTT can also send a text message but the free tier only allows a limited number of texts to be sent each month. I chose to use notifications through their Android app since they are unlimited and I would have burned through the text quota the first time I forgot to limit how often I send a notification. In a live situation, it could send out multiple texts.

Challenges

I had intended to use OpenHab for the home automation features of the project, but when I built the project there was a bug in the Z-Wave addon that made interacting with the deadbolt more difficult. I tried out Home-Assistant.io and so far I really like it. Each application has its own strengths and weaknesses, but they both run on a Raspberry Pi so I may use both for future projects. I’d like to mention they are both open source which is an added bonus.

The beacon distance tracking is not as accurate as I hoped, but it’s fine for this project. The signal can be degraded by walls, bodies or other objects being between the beacon and scanner. To improve the accuracy, I implemented a weighted rolling average function as part of the distance calculation to smooth out some of the spikes. Deploying more scanners would also greatly improve the accuracy if I implement the position tracking.

Future Improvements

Add an Oracle JET front end for configuration and control of the system.
Add a map display that can show the beacons live.
Change the Alert/Action code to be more generic and provide a mechanism to define them in the front end.
Find a small inexpensive wearable BLE beacon or design one with a small rechargeable battery and a 3D printed enclosure.

Final Thoughts

If you decided not to participate in the GeekAThon this year please join in next year. It is a great way to learn some new skills and have fun at the same time. I am sure parts of what I described above sound intimidating. But if you’d like to try your hand at this or similar projects, don’t hesitate to contact me for help. And while I can’t speak for the other GeekAThon participants, this year or past years, I am certain they will be eager to help you, too.

This project has been a lot of fun, I learned a lot. I’m looking forward to next year!

Information, JET, Meetups/Conventions, Python

Track-a-Watt – IoT to the Database: code walkthrough

June 8, 2017 bcarter Leave a comment

This is a companion post to my Track-a-Watt – IoT to the Database presentation.

If I missed you at GLOC 2017, you can still catch it at KScope 2017 or OpenWest 2017.

I’ve packed loads of stuff into this presentation, including: soldering (no software involved), Python, Javascript, HTML, PL/SQL and a little SQL (there has to be at least a little SQL in any application! :-)).

Even if I had a few hours of presentation time, it’d be hard to do justice to all these different scripts in their different languages, without losing lots of my audience somewhere along the way. So the presentation keeps things brief and to the point, and I will use this post to provide more depth for some of the code sections.

Python modules

sensorhistory.py

I mention that there are some names and labels used in this module that reference “5 minutes”.

I didn’t find any instances where a value for 5 minutes (300 seconds) is used in the functionality. Five minutes is only used as labels and object names.

The declarations for these can be found on lines:

103 – cumulative5mwatthr.
A variable used to store the cumulative watts per hour readings since the timer was started. We’ll call this total-watt-hours below.
105 – fiveminutetimer.
A variable used to store the time when the timer was initialized. We’ll call this start-time below.
119 – reset5mintimer.
A function to reset start-time and total-watts.
123 – avgwattover5min.
A function that prints the current data and returns the calculated average watts per hour since the timer started.
124 – fivetimer.
A text label in the print statement.
125 – 5mintimer and 5minwatthr
Labels in the text returned by the __str__ function.

This is just a demo, so I didn’t rename these objects. I only highlight these in case the names cause confusion after I change the timer to 10 seconds.

xbee.py

I only made one change in this module due to an error I received. I have been running this project on both Windows 7 and Fedora 25 machines. On one machine the values for p are passed in as Unicode and the other they are Strings.

The change here just checks to see if p is a String if so, convert it to Unicode otherwise accept it as is. Thanks, Anthony Tuininga for making this clean and compact.

def init_with_packet(self, p):
    #   p = [ord(c) for c in p]
        p = [ord(c) for c in p] if isinstance(p, str) else [c for c in p]

def init_with_packet(self, p):

# p = [ord(c) for c in p]

p = [ord(c) for c in p] if isinstance(p, str) else [c for c in p]

wattcher.py to wattcher-min.py

The original code for the Tweet-a-Watt project has some functionality that I don’t intend to use for my simple graph. I created the wattcher-min.py module by stripping out most of these features.

Average Watts/Hour calculation

As far as I can prove with my (cough cough) math skills, the algorithm used to calculate watts per hour works for whatever time slice you want to track.

I have not gone through all of the code that leads up to this point, but as I understand it:

The kill-o-watt is collecting a constant stream of readings.
The kill-o-watt X-Bee transmits the reading to the computer every 2 seconds where the data is stored in the array, wattdata[].
This code calculates and stores the average watts used in the last second.

# sum up power drawn over one 1/60hz cycle
avgwatt = 0
# 16.6 samples per second, one cycle = ~17 samples
for i in range(17):
    avgwatt += abs(wattdata[i])
avgwatt /= 17.0

127

128

129

130

131

132

# sum up power drawn over one 1/60hz cycle

avgwatt = 0

# 16.6 samples per second, one cycle = ~17 samples

for i in range(17):

avgwatt += abs(wattdata[i])

avgwatt /= 17.0

To calculate the average W/Hr during our current time slice:

Calculate the number of seconds since the last reading.
Multiply the average watts per second by the elapsed seconds then divide by 3600 (seconds in an hour).
Reset the last reading timer.
Print the data.
Add the calculated average W/Hr for this time slice to the running total.

# add up the delta-watthr used since last reading
# Figure out how many watt hours were used since last reading
elapsedseconds = time.time() - sensorhistory.lasttime
dwatthr = (avgwatt * elapsedseconds) / (60.0 * 60.0)  # 60 seconds in 60 minutes = 1 hr
sensorhistory.lasttime = time.time()
print("\t\tWh used in last {} seconds: {}".format(elapsedseconds,dwatthr))
sensorhistory.addwatthr(dwatthr)

146

147

148

149

150

151

152

# add up the delta-watthr used since last reading

# Figure out how many watt hours were used since last reading

elapsedseconds = time.time() - sensorhistory.lasttime

dwatthr = (avgwatt * elapsedseconds) / (60.0 * 60.0) # 60 seconds in 60 minutes = 1 hr

sensorhistory.lasttime = time.time()

print("\t\tWh used in last {} seconds: {}".format(elapsedseconds,dwatthr))

sensorhistory.addwatthr(dwatthr)

Here’s a basic explanation:

When a chunk of data comes in, we calculate the average W/Hr for the first second of that chunk. Multiply that value by the number of seconds since the previous reading. This gives us the average W/Hr for a 2 second time slice. If we were to collect those slices for one hour and add them together we would have X watts used in one hour.

The cumulative watts used will continue to accrue until we pass the limit of the timer we’re using to determine how often to send the data up to ORDS.

# Determine the minute of the hour (ie 6:42 -> '42')
currminute = (int(time.time())/60) % 10
print(int(time.time()))
# Figure out if its been five minutes since our last save
if (((time.time() - sensorhistory.fiveminutetimer) >= 60.0)
    and (currminute % 5 == 0)
    ):

154

155

156

157

158

159

160

# Determine the minute of the hour (ie 6:42 -> '42')

currminute = (int(time.time())/60) % 10

print(int(time.time()))

# Figure out if its been five minutes since our last save

if (((time.time() - sensorhistory.fiveminutetimer) >= 60.0)

and (currminute % 5 == 0)

# Determine the minute of the hour (ie 6:42 -> '42')
currminute = (int(time.time())/60) % 10
print(int(time.time()))
# Figure out if its been five minutes since our last save
if (((time.time() - sensorhistory.fiveminutetimer) >= 10.0)
   # and (currminute % 5 == 0)
    ):

154

155

156

157

158

159

160

# Determine the minute of the hour (ie 6:42 -> '42')

currminute = (int(time.time())/60) % 10

print(int(time.time()))

# Figure out if its been five minutes since our last save

if (((time.time() - sensorhistory.fiveminutetimer) >= 10.0)

# and (currminute % 5 == 0)

To calculate the average W/Hr during the last 10 seconds:

Multiply the cumulative watts used by 3600 (seconds in an hour).
Divide by the seconds since the last time we sent data to ORDS.

def avgwattover5min(self):
    print("cumulative: %f, time.time: %f, fivetimer: %f" % (self.cumulative5mwatthr, time.time(), self.fiveminutetimer))
    return self.cumulative5mwatthr * (60.0*60.0 / (time.time() - self.fiveminutetimer))

123

124

125

def avgwattover5min(self):

print("cumulative: %f, time.time: %f, fivetimer: %f" % (self.cumulative5mwatthr, time.time(), self.fiveminutetimer))

return self.cumulative5mwatthr * (60.0*60.0 / (time.time() - self.fiveminutetimer))

The short explanation is if we were getting a consistent reading of 5 watts per hour for every sample, every 10 seconds this calculation would come out to 5 W/Hr during the last 10 seconds. However, it’s not likely that we will get the same 5 W/Hr every reading so this function will give us the average W/Hr during the last 10 seconds.

I can understand if you’re a bit confused at this point. There seem to be a couple extra steps here than what should be needed for my simple graph. I had to work out a simulation in a spreadsheet before I could accept that it was working. However, I left the calculation code alone assuming that it may be needed for some of the more advanced versions of the project.

If your math skills are better than mine and you find that my explanation is wrong or you can explain it better, please leave a comment.

Oracle Jet

The Oracle Jet graph used in the example is the basic Line with Area Chart. I’m using the Y axis for the W/Hr data and the X axis for the timestamps.

The graph has the capability to track multiple series of data which would be useful for multiple kill-a-watts, but I’m only using one in the presentation.

The relationship between the X and Y axises is positional using the array position for the data elements in two arrays.

JavaScript

This is a typical jQuery ajax GET function.

Inside the success function:

Get the items array from the response.
Create a variable for the X-axis data.
Create a variable for the Y-axis data. Since we’re only tracking one sensor we can define the name value and initialize an items array for it.
Loop through the response items.
Populate the items array for our sensor (Y axis).
Populate the timestamp array (X axis).
Set the ko.observable objects for the two axises of the graph.

Next is a short function to call getData() every 5 seconds.

var getData = function() {
  $.ajax({
    type: "GET",
    url: 'https://oda-a425020.db.us2.oraclecloudapps.com/apex/kaw_demo/power/usage',
    success: function(res) {
      //Get the items from the response
      var resData = res.items;
      var groupData = [];
      var areaSeries = [{name: "Sensor 1", items:[]}];

      for (var i = 0; i < resData.length; i++) {
        //push the watt value to the areaSeries[0] items array
        //  We are only tracking one sensor so it's always areaSeries[0]
        areaSeries[0].items.push(resData[i].watt);

        //push the created_on value to the groupData array
        groupData.push(resData[i].created_on);
      }

      //set the observables to the proper values
      self.areaSeriesValue(areaSeries);
      self.areaGroupsValue(groupData);
    },
    failure: function(jqXHR, textStatus, errorThrown) {
      alert(textStatus);
    }
  });
};

//Get the data every 5 seconds.
setInterval(function() {
  getData();
}, 5000);

var getData = function() {

$.ajax({

type: "GET",

url: 'https://oda-a425020.db.us2.oraclecloudapps.com/apex/kaw_demo/power/usage',

success: function(res) {

//Get the items from the response

var resData = res.items;

var groupData = [];

var areaSeries = [{name: "Sensor 1", items:[]}];

for (var i = 0; i < resData.length; i++) {

//push the watt value to the areaSeries[0] items array

// We are only tracking one sensor so it's always areaSeries[0]

areaSeries[0].items.push(resData[i].watt);

//push the created_on value to the groupData array

groupData.push(resData[i].created_on);

}

//set the observables to the proper values

self.areaSeriesValue(areaSeries);

self.areaGroupsValue(groupData);

failure: function(jqXHR, textStatus, errorThrown) {

alert(textStatus);

}

});

};

//Get the data every 5 seconds.

setInterval(function() {

getData();

}, 5000);

HTML

We copy the HTML from the cookbook for just the graph component.

Since we’re not using the additional functionality from the Jet Cookbook example we remove the highlighted lines (14, 15).

<div id="lineAreaChart" data-bind="ojComponent: {
        component: 'ojChart',
        type: 'lineWithArea',
        series: areaSeriesValue,
        groups: areaGroupsValue,
        animationOnDisplay: 'auto',
        animationOnDataChange: 'auto',
        orientation: orientationValue,
        stack: stackValue,
        hoverBehavior: 'dim'
     }"
     style="max-width:1000px;width:100%;height:350px;">
</div>

<div id="lineAreaChart" data-bind="ojComponent: {

component: 'ojChart',

type: 'lineWithArea',

series: areaSeriesValue,

groups: areaGroupsValue,

animationOnDisplay: 'auto',

animationOnDataChange: 'auto',

orientation: orientationValue,

stack: stackValue,

hoverBehavior: 'dim'

style="max-width:1000px;width:100%;height:350px;">

</div>

Go try something new

The goal of this presentation is to encourage people to go out and try something a little out of their comfort zone. If you think your soldering skills are lacking find a maker group in your area and take a class. If you are strong in one programming language try another.

This is a great project to experiment with, there are a few different skills all mixed together, each of them is fairly close to entry level and they are popular enough that there should be a lot of help available.

As always, if you run into issues feel free to leave a comment here or hit me up on twitter and I’ll be glad to help get you going.

I plan to update this post as questions arise. If you’d like to see it all running together catch one of my upcoming sessions.

Learning, Meetups/Conventions

Leave your comfort zone behind: head to an open conference

July 20, 2016 bcarter Leave a comment

If you are a(n) programmer, you probably attend a(n) conference when you’re able. That’s great, since conferences are an awesome way to improve your skills in .

However, I also recommend that you try to attend an “open (source, hardware, data, ….)” conference if and when you can. The broader the scope, the better. Odds are, there will still be something related to your skills that you will find useful (and can use as justification to attend, if need be) but the real value comes from semi-random association with new stuff. Stuff you’ve never seen before, maybe never thought about before.

Recently, I attended OpenWest and of course there were plenty of sessions on technologies I use regularly (NodeJS, Python, Raspberry Pi, Docker, Privacy/Security, and a ton of stuff I don’t know how to do …. yet). I attended a few of these and learned some new tricks. I also attended some sessions that were out of my comfort zone. I learned a bit about Bitcoin and Block Chain, tried to keep up with some Linux server discussions, was completely out of my league listening to a session on Recent Advances in Microcontrollers and, after struggling through some basic “hacker” challenges, I got to dust off my soldering skills and make a cool LED badge.

Why does this matter? Diversity is the key to sustained growth. This famously applies to your financial investments, but it’s also important to your growth as a technologist. If you continue to do the same/similar thing over and over, with the same technology, you will undoubtedly get really good at it. The potential downside, however, is that you may be perceived as that person who only has a hammer, so everything looks like a nail.

If you enjoy being the go-to person when there’s a problem to be solved in technology X and no expert in X in sight, you need to keep adding to your bag of tricks. You need to keep challenging yourself.

This doesn’t mean that you have to have all of the answers. Sometimes it’s enough to know that a better solution exists “out there” somewhere, and help get the team pointed in the right direction.

Before going to OpenWest, I had experimented with my Raspberry Pi, but only in software ways like OSMC and OpenHab. I do have an Arduino and I’ve read through a few projects that look fun, but I’ve hesitated to get started, assuming that it’s going to take too much time to ramp up my skills and I’ll probably burn it up unless I practice my soldering skills…. a lot.

Now, I’ve learned that the hardware components are more rugged than I assumed and I’m not going to burn them up with a minor solder error and once again I’m experiencing with how much fun it is to be the “new kid” learning cool, new things.

It is definitely time to go buy a good soldering iron.

Learning, Meetups/Conventions

Collaborate 15

April 20, 2015 bcarter Leave a comment

Last week I had the privileged to attend Collaborate 15 in Las Vegas.

The facilities at the Mandalay Bay were top notch. The food was very good and there were plenty of quiet comfortable places where you could catch your breath, relax and catch up.

I was only able to attend for a couple days at the end, but I still learned a lot and made a few great connections. I came away with new useful information from every session I attended, even the one I jumped into late just for a place to sit down. A couple of my favorite sessions had lively discussions that were focused, on topic and helpful.

I think most people prefer substance over flash when they chose a conference, at least I know I do, but some flash is fun. I really enjoyed the mix they put together. The floor exhibits were interesting and there were a lot of them. The 60s party was a blast and the band was awesome. Just when I would reach the “my brain is full” point, there was something shiny to focus on while my mind absorbed the info and I could relax before loading up the next chunk of information.

Overall I thought it was a great conference and I would definitely recommend it.

Problem

Proposal

Desired Features

Initial Idea

Hardware

Software

Database

How I Deployed the System

Challenges

Future Improvements

Final Thoughts

Share this:

Python modules

sensorhistory.py

xbee.py

wattcher.py to wattcher-min.py

Average Watts/Hour calculation

Oracle Jet

JavaScript

HTML

Go try something new

Share this:

Share this:

Share this: