Streaming Twitter with C#

In this article I will walk through the steps required to create a C# console application that prints a Twitter stream to the console using TweetInvi library

The example was built using Visual Studio 2015 Community Edition and .NET Framework 4.6.

Step 1

Start Visual Studio and create a new console application, I’ve called mine TwitterPublicStream.

 Step 2

Right click on the project in the solution explorer window (In the example below this is TwitterPublicStream) and select Manage Nuget Packages

 

 

 

 

 Step 3

Search for tweetinvi and once found, install it, accepting the various licences, if you are happy to do so.

 Step 4

In order to use the Twitter API’s, you first need to obtain some credentials. To do this you now need to visit the Twitter API home page and follow the instructions.

Step 5

After that 4 step ceremony we are now ready to write some code.

using System;
using Tweetinvi;

namespace TwitterPublicStream
{
 class Program
 {
 static void Main(string[] args)
 {
 // add your Twitter API credentials here 
 Auth.SetUserCredentials("CONSUMER_KEY", "CONSUMER_SECRET", "ACCESS_TOKEN", "ACCESS_TOKEN_SECRET");

 var stream = Stream.CreateFilteredStream();
 // change LEITOT to something that is currently trending on twitter 
 stream.AddTrack("LEITOT");
 stream.MatchingTweetReceived += (sender, theTweet) =>
 {
   
    Console.WriteLine($"A tweet containing LEITOT has been found; the tweet is {theTweet.Tweet}");
 
 };
 stream.StartStreamMatchingAllConditions();

 }
 }
}

At line 2 a using statement is added for the Tweetinvi library.

At line 11 you need to add your Twitter API credentials that you obtained in Step 4.

At line 15 the AddTrack method is called. Track in the Twitter API context is a comma-separated list of phrases which will be used to determine what Tweets will be delivered on the stream. You can find out more here. Whilst testing this I suggest selecting a trending topic without the #. The one shown in the code was a football game between Leicester and Spurs.

At line 16 the MatchingTweetReceived event will output the contents of the tweet to the console.

Line 21 starts the streaming.

Step 6

In the final step, compile and run the program. After a few seconds you should start seeing Tweets populate the console window.

 

Summary

In this article I have explained how to use the superb library TweetInvi to stream Tweets of interest from Twitter into a C# console application.

C# Utility to emulate the XPath 3 function path()

I recently needed to examine a number of XML files and print out the element names that contained text greater than X number of characters. In addition I also need to print the location of that element within the XML document.

i.e. given the following document

<bookshop>
  <book>
    <title>Microsoft Visual C# Step by Step</title>
  </book>
</bookshop>

…if I was interested in book titles that had more then 10 characters I would want to see:

/bookshop/book/title/
Microsoft Visual C# Step by Step

Whilst it is straightforward to return the text node, finding the XPATH location proved to be more challenging than I initially thought. The reason being is that whilst XPATH 3.0 introduced the path() function that returns the current XPATH location, the number of programming languages that I know (PL/SQL, Python and C#) do not implement XPATH 3.0 yet.

As a result I had to build my own utility. I chose to write this in C# as this is a language I have spent the past 18 months learning and I am now looking for real world problems I can solve using it.

The utility can be found on github. The “engine” of the utility is copied from this Stackoverflow answer: http://stackoverflow.com/a/241291/55640 provided by Jon Skeet.

Although far from feature complete I hope it will give someone facing a similar challenge a head start.

Let me know what you think.

Naming Windows .bat files

Whilst setting up a Elixir development environment I ran into a interesting problem.

I created a new .bat file in Windows 10 to start up Elixir’s Interactive Shell. The command I wanted to run was:

iex --werl

So I created a new Windows .bat file called iex.bat which contained this single command.

Unfortunately when I double clicked on the icon instead of seeing the Elixir Interactive Shell I saw that the command was continually looping:

 

 

 

 

 

 

 

Fortunately this Stackoverflow answer helped in identifying the problem. If you create a .bat file with the same name as the command you wish to run you will end up in a endless loop!

If this article has helped you, please take a moment to up vote the Stackoverflow answer.

 

You don’t have to suck at Excel

Watching Joel Spolsky’s masterful presentation You Suck at Excel with Joel Spolsky will make you better at using Excel.

You don’t have to watch it all , watch the first 5 minutes about how to correctly paste in Excel will improve your productivity whilst reducing your #REF! error stress levels and will put you head and shoulders above most of the people using Excel today.

Some other highlights to look out for in the first 20 minutes are:

  1.  R1C1 mode
  2. Riding the Range
  3. Rounding errors

I could go on and on but I am stopping you watching the video so suffice to say this is such an excellent and generous presentation by a superb technical leader.

Thank you Joel.

 

 

Book Review: Introduction to Javascript Object Notation by Lindsay Bassett

Whilst attending the UKOUG Tech16 conference, several of the talks I attended mentioned the use of JSON (or to give it’s full name JavaScript Object Notation) These talks made me realise how little I actually knew about this Data Interchange Format.

There are many resources for learning JSON, from websites of varying quality to paid for video courses on Pluralsight or for free on YouTube. However my favourite method of learning something new is by reading book(s) and then conducting experiments using what I have learnt. So I chose the following: Introduction to Javascript Object Notation by Lindsay Bassett.  I chose this title as it had a good review and at just over 100 pages was not going to be a door stop that I will never finish.

The book begins with an overview of JSON, it’s syntax, the available datatypes and validating your JSON documents using JSON Schema before switching gears and moving on to demonstrate how JSON can be used in client and server side frameworks and NoSQL databases.

The book was a pleasure to read, new concepts are concisely introduced and no assumptions of your knowledge are made and having now read it I am far more confident in my understanding of this latest Interchange Format.

If you are looking to get up and running with JSON it is easy for me to recommend this book.

 

 

 

Technical Books I have read in 2016

I have always enjoyed reading books about Programming. From books that lead you to take your first tentative steps with a new language to ones that take you on a deep dive into the world of particular feature. I especially enjoy ones that discuss language agnostic programming concepts such as debugging, estimating etc. Books like Code Complete, The Pragmatic Programmers, The Mythical Man Month and Don’t Make Me Think.

To me technical books are such a bargain. For £20 – £30 you can gain knowledge and insight that can make you so much better at your job, such as taking different approaches to solving the daily problems that we as programmers face. Without a doubt there is a lot of published rubbish out there but fortunately in these days of reviews and questions on the numerous Stack Exchange sites it is a lot easier to avoid the charlatans and their ammo pouches stuffed with silver bullets. Although as you will see from my own list, one or two may still slip through the net!

Here are the programming related books I have read this year, listed in the order that they were read.

cplayersThe C# Player’s Guide (2nd Edition)

This is my favourite book that I have read whilst learning C#. Immediately accessible. The large format of the book along with the lucid and easy to grasp descriptions of Object Orientated topics make this my recommended book to anyone that is interested in learning C#.

Django By Example djangobe

Unfortunately this book is still on the “bought but not read” pile. It is no reflection on the book I have been focusing my attention on learning C# this year.

C# 6.0 and the .NET 4.6cnet46 Framework

At 1600+ pages this was certainly the biggest technical book I bought this year. For me it is too unwieldy to use on a day to day basis so, for the first time I have abandoned the printed version of a book and have spent the last 8 months using the e-book. Usually the ebook is open on one monitor whilst Visual Studio is open in the other. Not sure if it’s such a good book for beginners but as a reference I can see myself returning to it to look things up.

The Psychology of Computer Programming: Silver Anniversary Editionpcp

I have been wanting to read this book for several years and finally got round to it. It is by a very long way my favourite read this year and it is in the top 5 all time technical books I have ever read. Although 45 years old, the ideas discussed then are still very relevant today; How we don’t read existing code to see how others have solved problems, the critical importance of having code reviews, egoless programming, estimating and setting expectations around delivery times. I could go on and on. If you haven’t read it, order it today you will not regret it. It will make you a better programmer or manager!

learnciadLearn C# in One Day and learn it well

The worse book I read this year. I have already written what I think of it here.  Not much more to add so moving on to the final book…..

Working Effectively With Legacy Code wewlc

The final book for this year is another classic and I have high expectations for it. Currently I am a third of a way through but I will have finished it by the end of the year. At this point I think it should be called “Working Effectively with Legacy Object Oriented Code” because a lot of the ideas in the book code are centred around legacy Object Oriented code. I will update this once I get to the end of the book.

Summary

This year marks a slight change from previous year lists in that I haven’t read any Oracle database or Application Express books. There are two reasons for this. First I don’t think there have been any unmissable Oracle books published this year (I am interested in Real World SQL and PL/SQL that was published in September 2016 however I awaiting reviews or to actually have a look through it) –  and secondly most of my spare time has been spent learning C#.

I have taken something from each of these five books this year, yes even Learn C# in a day. I know that as a result of reading these books, I will start 2017 a better programmer.

An introduction to Web scraping using Python 3

In this article I will demonstrate how easy it is to perform basic text Web scraping using Python and just a few lines of code.

The example have been developed and tested using Python  3.5.2.

The first step is to see if you have the following third party libraries already installed; Requests and Beautiful Soup 4. So start idle and try typing the following command:


import requests

After you press return, if you see no error messages then requests is installed. If you see an error message that shows requests has not been found, you should install it using pip from the command line as shown below.


pip install requests

Repeat the process to see if you already have the Beautiful Soup library installed, fortunately you don’t have too much to type….


import bs4

Again if Python complains that it can’t find the library, use pip from the command line to install it.


pip install beautifulsoup4

With the libraries installed, here is a program that scrapes this site. It returns the titles from the blog posts that are shown on this page.

To demonstrate how this is achieved with just a few lines of code, here is the program without comments:


import requests, bs4

def getTitlesFromMySite(url):

 res = requests.get(url)
 res.raise_for_status()

 soup = bs4.BeautifulSoup(res.text, 'html.parser')
 elems = soup.select('.entry-title')
 
 return elems


titles = getTitlesFromMySite('http://www.oraclefrontovik.com')

for title in titles:
 print(title.text)

Now the same code but this time with each section commented…


# import requests (for downloading web pages) and beautiful soup (for parsing html) 
import requests, bs4

# create a function that allows a parameter containing a url to be passed into it
def getTitlesFromMySite(url):

# download the webpage and store it in res variable
res = requests.get(url)
# check for problems - if there are, raise_for_status() raises an exception
# and the program stops at this point
res.raise_for_status()

# running the downloaded webpage through Beautiful Soup returns a
# Beautiful Soup object which represents the HTML as a nested data structure.
soup = bs4.BeautifulSoup(res.text, 'html.parser')

# store in an array the items that match this css selector. 
# I will explain how I obtained this entry below
elems = soup.select('.entry-title')

return elems

# call the function and store the results in titles
titles = getTitlesFromMySite('http://www.oraclefrontovik.com')

# loop through the array printing out the title.
for title in titles:
print(title.text)

Running the example returns the following expected output….


Learn C# in One Day and Learn It Well – Review

Contributing to an Open Source Project

A step by step guide to building a Raspberry Pi Hedgehog camera

Is there more than one reason to use PL/SQL WHERE CURRENT OF ?

Structured Basis Testing

Raspberry Pi connected to WiFi but no internet access

The auditing capabilities of Flashback Data Archive in Oracle 12c.

DBMS_UTILITY.FORMAT_ERROR_BACKTRACE and the perils of the RAISE statement

Using INSERT ALL with related tables

The best lesson I learnt from Steve McConnell

To summarise, the code imports two third party libraries, requests and Beautiful Soup 4, that perform the lions share of the work. In the example I use the requests library to download a web page as HTML and then pass it to Beautiful Soup along with a CSS selector to return the information I want from it.

Obtaining the CSS selector

The code example has the following line which extracts the part of the webpage, the blog post titles, that we are interested in:

elems = soup.select('.entry-title')

Using Firefox, I obtained the CSS Selector ‘.entry-title’ by:

  1. Navigate to the page of interest, in this case, oraclefrontovik.com
  2. Opened Firefox developer tools (Ctrl + Shift + I)
  3. Highlighted the first title (which at the time of writing was Learn C# in One Day and Learn it Well – Review) , right click and select Inspect Element
  4. In the console, I then right click and select Copy and then choose CSS Selector from the sub menu.

At the time of writing, I was unable to get the same CSS Selector using the native developer tools from Chrome. If you know of a way please let me know in the comments.

Summary

In this post I have walked through the steps to perform basic text Web scraping using Python 3.

Learn C# in One Day and Learn It Well – Review

I have been learning C# and the .NET framework for a while now and have been working my way through several books; The C# Programming Yellow Book, The C# Player’s Guide (2nd Edition) and C# 6.0 and the .NET 4.6 Framework All of these books have helped me to varying degrees to get comfortable in Object Oriented programming, the C# language and the .NET framework.

When learning a new programming language, I always look to improve my knowledge of the fundamentals, so seeing an introduction to C# book that was getting good reviews piqued my interest. That book was Learn C# in One Day and Learn It Well although I am very suspicious of Learn X in Y days\hours\minutes titles (see Peter Novig masterly description) I ordered a copy.

learncinaday

 

 

 

 

 

 

 

At 153 pages the book is slim and can be divided into two parts. Chapters 1 through 11 cover the various building blocks that make up a programming language such as variables, arrays, condition statements as well as briefly touching on Object Orientated concepts. The second half of the book, starting on page 128 brings together what you have learnt in a project by building a Payroll programme.

I think the book is self published, obviously not an issue in itself however I felt that it could have done with a review\editor to catch the typos and misaligned paragraphs but these are minor irritants. The real point of this post is, can you use this book to learn C# in a Day?

In my opinion no. The main problem with this book is how brief the topics are covered. Take for example Interfaces which are discussed on pages 107 – 109. The text compares Interfaces with Abstract classes, however no where in these two pages does it tell you what an interface actually is and why you would want to create one.

In summary I am not sure who the target audience for this book is. Perhaps someone that just needs to get some course work or module “working” For everyone else it is far to brief and does not go into enough detail especially explaining why you would want to use a feature of the language. If you are interested in learning C# my advice would be to put the £8 towards a better resource.

 

 

Contributing to an Open Source Project

I have been interested in Git, the distributed version control software since reading the first edition of Scott Chacon’s Git book way back in 2010. However outside of my own projects, my real world experience of using Git is relatively limited and it’s one of the skills I never seem to get around to improving on.

To change this, I have recently contributed to an open source project hosted on GitHub. The change I made can be found here and this post is my recollection of the process to help me and hopefully others just getting started with the GitHub workflow.

For a comprehensive guide to the GitHub workflow, I recommend reading Chapter 6 of the Git book.

Find a project that you want to contribute to.

Probably the most tricky step – there are so many projects how do you find one to contribute to? In my case I started with a project that I know and use. OraOpenSource/Logger which is a great tool for instrumenting Oracle PL/SQL code.

githubissues

 

From there it’s a quick scan of the open issues. I picked one related to the documentation because I wanted to focus on the GitHub workflow. The challenging technical issues and enhancements will still be there once I have got up to speed with the GitHub way of working.

Once you have found a project, it is unlikely that you will be able to push your changes to it, so the next step is to fork it. This gives you a copy of the project within your users namespace which you can then make changes to.

Make the change

With the project forked, you can go ahead, create a topic branch and make the necessary changes to the files and once you are happy with them, push them back to your copy of the project.

Pull request and …..Oops!

When you are ready to contribute your changes back to the original project you need to create a pull request. Creating a pull request opens up a discussion thread with a code review focusing on your proposed change.

Don’t worry if the change is discussed or rejected. Dust yourself down and go again. I had my own oops moment with my first pull request as I had changed a URL from relative to absolute. Not a problem so I closed the initial pull request and created another which has now been accepted and merged into the project.

Make the world a better place

Apologies the for the heading, I have been enjoying Silicon Valley around the time this post was taking shape.  If not the world, your change no matter how small will make the project you are contributing to better and it gives you a public artefact that you can point to.

Summary

In this article I had written about my Git experience along with my first contribution to an open source project.

 

A step by step guide to building a Raspberry Pi Hedgehog camera

 

What the camera was designed to do! Photograph of a Hedgehog in the garden

I had been looking for an Raspberry Pi project for a long time, the various media player and game emulators didn’t interest me enough besides I wanted something that the whole family could get involved in.  It was only through discussing this at work that someone mentioned they had built a Hedgehog camera with a Raspberry Pi. Awesome! That was exactly the project I was looking for.

So over the course of several nights I built a prototype and was genuinely surprised at a) how much fun I had putting it together, b) that it actually worked and finally c) how much the whole family has got involved.

One thing I found pretty much straight away was although there is plenty of information on the web to build one of these devices it is not all in one place and I found myself going from one site to another so this post is my attempt to bring all the information and what I learnt whilst building it into one place. I will of course acknowledge which sites helped.

Before going any further here is what my device looks like.

Ugly but effective! The completed Hedgehog camera

 

Granted, it won’t win any design awards. You can spend more time making it look far more aesthetically pleasing if you so wish.

What you will need and how much it will cost

A Raspberry PI (I am using the Raspberry Pi 3 and bought this starter kit)£41.50
A Raspberry PI Case. Unfortunately the official case that comes with the kit above requires you to remove the top part in order to fit the camera and as the PI will be spending a lot of it’s time outside I wanted to protect it as much as I could so I bought this case which has a slot for the camera ribbon£4.50
Waveshare Raspberry Pi Camera Night Vision Camera Module Kit£23.99
An air tight storage box.From 99p
A drill bit to make a hole in the storage box for the camera. After managing to destroy several boxes by using the wrong tools, I ordered this drill bit set which made the task a breeze.£7.95
A Battery Pack. I use this one but at the time of this post the cost has gone up considerably since I bought mine. You just need one that has a large capacity, the one I am using has an advertised capacity of 20100mAh and comfortably powers the camera all night.Around £25 - £30
Total:£109.44

Setting up the Raspberry Pi

If you are using an existing Raspberry PI and its software is up to date , then you can skip this step.  Out of the box my Raspberry PI required various updates. Using the advice from here  I ran

sudo apt-get update

to update the package list followed by

sudo apt-get dist-upgrade

to upgrade all packages to their latest version.

If you are going to use the default user, Pi remember to change the password!

Fitting the camera and installing the camera software

It is important to note that the Waveshare camera is an all in one device. By which I mean it comes with everything needed to take pictures at night time. You don’t need extra infra red led’s or motion sensors. I should also point out that the camera takes stills and not videos. What you can do using the software that I mention below is to “stitch” all the pictures taken in one night together to create a time lapse MP4 file that you can watch, upload to YouTube etc.

First you need to fit the camera to the Raspberry Pi. This video is an excellent tutorial, even though the camera module in the video is different, the installation is exactly the same.

With the camera now fitted you need to install software to control the camera and when pictures are taken. I chose pi-timolo.  This excellent piece of software can be configured to take photo’s whenever motion is detected. The installation is straight forward and as my Raspberry Pi is only being used for the Hedgehog camera I also followed the instructions to run on boot. Which, in case of need means that when the Raspberry Pi starts up, the camera starts capturing images once it detects motion.

The default location for the photos are:

/home/&lt;&lt;username&gt;&gt;/pi-timolo/motion

where <<username>> is the name of the user that installed pi-timolo.

At this point, we now have a Raspberry Pi camera that will take a photograph whenever it detects motion which means that the number of photographs taken every night will be measured in the hundreds. Sifting through these every day will very quickly lose it’s appeal (if it ever had one) so the final software package we are going to install is FFmpeg. This will “stitch” all the photographs for a day (or folder) into one MP4 file. Playing the MP4 file will give you a time lapse like experience. You can download FFmpeg from here. I will show you how to use FFmpeg in the Reviewing the Results section.

Fitting the PI and camera into the air tight container

Overview of the components within the storage box

Firstly make a guide in the middle of the airtight container. This should be at one end of container. Then using the drill bit I have suggested or another tool drill\cut a circle so that the camera can be fitted. You do not need to cut similar holes for the infra red sensors.  Then fix the camera to the inside of the container. I have used parcel tape(!) to secure mine. Not elegant but it works.

If you have not done so, now is the time to connect the camera to the Raspberry Pi.

The final step is to connect the battery to the Raspberry Pi, I normally do this just before I place the container in the garden.

The parcel tape around the outside of the box is to cut down the amount of light that is visible from within the box.

Where to place in the garden

The obvious answer is to place the camera in the area where you have seen Hedgehogs or their poo! However the key is to experiment so try various places around your garden.

Reviewing the Results

After collecting the container the following morning,  my workflow for reviewing the results is to first power down the Raspberry Pi (it is still using the battery at this point) and then connect it to the mains and restart it.

After navigating to

/home/&lt;&lt;username&gt;&gt;/pi-timolo/motion

I copy the photos to another folder within my home directory called

/home/&lt;&lt;username&gt;&gt;/garden/&lt;&lt;date_of_photos&gt;&gt;

and using the following FFmpeg command

ffmpeg -f image2 -pattern_type glob -i '*.jpg' 20160821.mp4

which creates an MP4 file from all the jpg files within the directory where this command was run.

After a few minutes the process will be complete and you will be able to play the MP4 file using your favourite player. When viewing on the Raspberry Pi I use Omxplayer  or other devices I use VLC.

Optional Extras

One optional extra that springs to mind is to add a WiFi extender dongle (if your garden is large enough) this will enable you to connect to the device and see what is happening in real time.

If you think of other optional extras please let me know via the comments.

Summary

In this article I have gone through the steps required to create a Hedgehog camera using a Raspberry Pi.

This write up has been made a couple of weeks after I had built my own version, so if I have missed something or I haven’t made a particular point clear enough please let me know in the comments.

Acknowledgements

My gratitude for a large part of this work, in particular the genius of using the air tight container goes to http://www.sconemad.com/blog/hedgeycam/