Using C# to upload a file to AWS S3 Part 2: Creating the C# Console App

This post follows on from part 1.  With the AWS S3 objects in place it is now time to create a simple C# console application that will upload a text file stored locally to the AWS S3 bucket.

The first step is to create a test file that you want to upload. In my example, I have created a text file in the Downloads folder called TheFile.txt which contains some text. After creating the text file, note the name of the file and its location.

Start Visual Studio and create a new console application

Use NuGet to add the AWSSDK.S3 package. At the time of writing this was at version 3.3.16.2

Add the following to App.config

 <appSettings>
   <add key="AWSProfileName" value="dotnet">
   <add key="AWSAccessKey" value="YOUR ACCESS KEY">
   <add key="AWSSecretKey" value="YOUR SECRET KEY">
 </appSettings>

You will find the values for the access key and secret key in the accessKeys.csv which you downloaded in part one of the tutorial.

Create a new class called S3Uploader and paste the following code ensuring you change the variables for bucketName, keyName and filePath as appropriate. As you can see from the comments, this code is based on this answer from Stack Overflow.

For the sake of brevity the code deliberately does not have any exception handling nor unit tests as I wanted this example to focus purely on the AWS API without any other distractions.

using Amazon.S3;
using Amazon.S3.Model;

namespace S3FileUploaderGeekOut
{

  /// <summary>
  /// Based upon https://stackoverflow.com/a/41382560/55640
  /// </summary>
  public class S3Uploader
  {
    private string bucketName = "myimportantfiles";
    private string keyName = "TheFile.txt";
    private string filePath = @"C:\Users\Ian\Downloads\TheFile.txt";

    public void UploadFile()
    {
      var client = new AmazonS3Client(Amazon.RegionEndpoint.EUWest2);
      
      PutObjectRequest putRequest = new PutObjectRequest
      {
        BucketName = bucketName,
        Key = keyName,
        FilePath = filePath,
        ContentType = "text/plain"
      };

      PutObjectResponse response = client.PutObject(putRequest);
    }
  }
} 

In the Program.cs class add the following:

namespace S3FileUploaderGeekOut
{
  class Program
  {
    static void Main(string[] args)
    {
      S3Uploader s3 = new S3Uploader();

      s3.UploadFile();
    }
  }
}

Run the program and once it completes, navigate to your S3  Bucket via the AWS console and you will be able to see that your file has been successfully uploaded.

Summary

In this and the previous post I have demonstrated the steps required to upload a text file from a simple C# console application to a AWS bucket.

Using C# to upload a file to AWS S3 Part 1: Creating and Securing your S3 Bucket

In this, the first of a two part post, I will show you how to upload a file to the Amazon Web Services (AWS) Simple Storage Service (S3 ) using a C# console application.

The goal of this post is to get a very simple example up and running with the minimum of friction. It not a deep dive into AWS S3 but a starting point which you can take in a direction of your choosing.

This post will focus on how to set up and secure your AWS S3 bucket.  Whilst the next will concentrate on the C# console app that will upload the file.

Dependencies

In order to build the demo the following items were used:

An AWS account. (I used the  12 months free tier)

Visual Studio 2017 Community Edition 

AWS Toolkit for Visual Studio 2017

Creating a new AWS S3 bucket

Log on to your AWS Management Console and select S3 (which can be found by using the search bar or looking under the Storage subheading)  

You should now be on the Amazon S3 page as shown below.

This page give you the headline features about your existing buckets. In the screenshot you can see an existing bucket along with various attributes.

Click the blue Create bucket button and enter a name for your bucket, the region where you wish to store your files and then click next.

Click Next.  This screen allows you to set various bucket properties. For this demo, I will not be setting any so click Next to move onto step 3

Leave the default permissions as they are and click Next to move on to the final page.

After reviewing the summary, click Create Bucket

IAM User, Group and Policy

In order to access the S3 bucket from the .NET  application valid AWS credentials are required. Whilst you could use the AWS account holders credentials, Amazon recommends creating an IAM user in order to utilise the IAM users credentials when invoking the AWS API.

In this section of the post I will show you how to create a new IAM user and give it just enough privileges required to interact with our new S3 bucket. The information shown below has been distilled from the AWS documentation.

There are a large number of steps that follow and it is easy to get lost. My advice is to read through once before diving in. If you get stuck (or I have missed something) let me know in the comments.

Return to the AWS Home screen

Select\search for IAM, and after selecting users on the left hand side menu, click the blue Add User Button which will bring up the Set user details page.

Give the user a name and the access type to Programmatic access only. There is no need for this user to be given access to the AWS console.  Click Next Permissions.

Rather than give permissions directly to the IAM user, Amazon recommends  that the user be placed in a group and manage permissions through policies that are attached to those groups. So lets do that now.

From the Set permissions page click on Create Group.

Give your Group a meaningful name.

The next step is to attach one or more policies to the group.  Policies in this context defines the permissions for the group. The Create group page lists the available policies but unfortunately there isn’t an existing policy that can be used to ensure that the IAM user has only access to the new S3 bucket, so click on the Create policy button.

This opens in a new browser tab, the Create policy page

Click on the JSON tab and copy the following. Changing the bucket name as appropriate.  (The source of this JSON can be found here.)

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Action": [
    "s3:ListAllMyBuckets"
     ],
     "Resource": "arn:aws:s3:::*"
  },
  {
   "Effect": "Allow",
   "Action": [
   "s3:ListBucket",
   "s3:GetBucketLocation"
   ],
   "Resource": "arn:aws:s3:::myimportantfiles"
  },
  {
   "Effect": "Allow",
   "Action": [
   "s3:PutObject",
   "s3:GetObject",
   "s3:DeleteObject"
   ],
   "Resource": "arn:aws:s3:::myimportantfiles/*"
   }
  ]
}

At this point the JSON editor should look like this.

Once done click on the Review policy button. Give your policy a meaningful name and description and then click Create policy.

You will then receive confirmation that the policy has been created.

Now click the browser tab which displays the Create group page.

To find your new policy, change the filter (located left of the search bar) to “Customer managed” and press the refresh button (located next to the Create policy button). Once you have found the newly created policy, select it and press the Create group button.

You will now be returned to the Set Permissions Page; ensure the new group is selected and click Next: Review.

The final page is a review after which you can then click Create user.

Once the user has been created, you will see a confirmation along with a download .csv button. Click the button to download the credentials as these will be needed in our C# application discussed in the next post.

 

 

 

Review

At this point it is worth getting a cup of your favourite beverage and recapping what has been created:

A new AWS S3 bucket.

A new IAM user. This user has been placed in a group. The group has a policy attached that allows it to perform various operations only on the new bucket that has been created.

A csv file containing the required access and secret keys have been downloaded.

On to part 2

With the S3 bucket and IAM user and the necessary privileges created and configured it is time to move on to part two which will create the .NET console application to upload a file into this bucket.

Visual Studio’s Live Share != Screen Share

I am very excited about the potential of Live Share since it’s announcement back in November.

In addition to the content on the Live Share site, I would also recommend listening to this episode of Hanselminutes where Scott talks to Amanda Silver about how Visual Studio’s Live Share goes far beyond “text editor sharing” to something deeply technically interesting.

Although still very early days, Live Share is definitely something worth keeping an eye on. A private preview is coming soon and you can sign up for for more information here.

C# Attributes

Overview

This post gives an overview of C# attributes; what they are, the different flavours, when they should be used and finishing up with some examples.

What are C# Attributes?

Attributes in C# provide a mechanism of associating Metadata with code entities such as types, methods, properties etc. Once defined you can then retrieve the Metadata at run time using Reflection.

What flavours do they come in?

Two flavours of attributes are available:

  1. Predefined which are the ones provided by the .NET base class library
  2. Custom which are created by the developer to fill in any gaps in the required Metadata.

Attributes can be defined on many different targets; such as the assembly, type (Struct, class interface), methods etc. Please refer to the table here for the full list of targets available.

When should I use them?

Reviewing the official .NET guide on Attributes for this post, the docs give a verbose introduction to Attributes. Myself I prefer the following quote that is taken from an blog post by Eric Lippert which succinctly explains the reason for using attributes

use attributes to describe your mechanisms

Example 1: The Obsolete Predefined Attribute

This first example shows how to use the Obsolete Predefined attribute.

The class Foo has a method that is now obsolete and should no longer be used.

using System;

namespace ObsoleteAttributeGeekOut
{
  class Foo
  {
    [Obsolete(message:"Use NewBar() instead")]
    public void OldBar()
    {
      Console.WriteLine("Calling OldBar");
    }

    public void NewBar()
    {
      Console.WriteLine("Calling NewBar");
    }
  }
}

I have added the Obsolete Attribute to the OldFoo method along with a message to help developers identify which method should be used instead.

If I then try to use OldBar, as in the code below

namespace ObsoleteAttributeGeekOut
{
  class Program
  {
    static void Main(string[] args)
    {
      Foo f = new Foo();
      f.OldBar();
    }
  }
}

Visual Studio displays a helpful message.

 

 

 

 

 

 

When compiling the program, you will see the following warning:

ObsoleteAttributeGeekOut\Program.cs(10,13,10,23): warning CS0618: ‘Foo.OldBar()’ is obsolete: ‘Use NewBar() instead’

The code for this example can be found here.

Example 2: Creating a Custom Class Attribute & Using Reflection to examine the data at run time

The first step to create a Custom Attribute is to create a new class deriving from System.Attribute as the code below demonstrates.

using System;

namespace ClassLevelAttributes
{
  [AttributeUsage(AttributeTargets.Class)]
  public sealed class UsefulClassMessageAttribute : Attribute
  {
    private string s;

    public string S
    {
      get
      {
        return s;
      }
    }

    public UsefulClassMessageAttribute(string s)
    {
      this.s = s;
    }
  }
}

There are several items of note that are worthy of further explanation about this class.

  1. I use a Predefined attribute to define the level that the Custom attribute should be used at.
  2. The class is sealed. Rather than sidetrack the example if you are interested in reading more on why you should seal your attribute classes take a look at this StackOverflow question.
  3. The name of the class ends with Attribute. This is a convention rather than a rule and is recommended for readability. As you will see when the attribute is applied the word Attribute is optional.

With the Custom Attribute, UsefulClassMessageAttribute defined I can now go ahead and use it as in the code below.

using System;

namespace ClassLevelAttributes
{
  [UsefulClassMessage(s:"This is something useful")]
  class Example
  {
    public void Foo()
    {
      Console.WriteLine("Called Foo");
    }
  }
}

Here you can see the Custom Attribute applied to the class Example. Note that the name used is UsefulClassMessage and not UsefulClassMessageAttribute.

A Custom attribute is worthless until you can see the information. So for the final code in this example I use the following method to obtain the Custom Attribute information.

using System;

namespace ClassLevelAttributes
{
  class Program
  {
    static void Main(string[] args)
    {
      GetAttribute(typeof(Example));
    }

    public static void GetAttribute(Type t)
    {
      UsefulClassMessageAttribute theAttribute = (UsefulClassMessageAttribute)Attribute.GetCustomAttribute(t, typeof(UsefulClassMessageAttribute));

      if(theAttribute == null)
      {
        Console.WriteLine("The attribute was not found");
      }
      else
      {
        Console.WriteLine($"The attribute value is: {theAttribute.S}");
      }
    }
  }
}

Running this, you would see the following in the console window:

The attribute value is: This is something useful

The code for this example can be found here.

Example 3: Creating a Custom Method Attribute & Using Reflection to examine the data at run time

As shown in Example 2, the first step in creating a Custom Method is to create a class that derives from System.Attribute.

using System;

namespace MethodLevelAttributes
{
  [AttributeUsage(AttributeTargets.Method)]
  public sealed class UsefulMethodMessageAttribute : Attribute
  {
    string s;
    public string S
    {
      get
      {
        return s;
      }
    }

    public UsefulMethodMessageAttribute(string s)
    {
      this.s = s;
    }
  }
}

With the exception of it’s name and that the AttributeUsage which is now set to Method, this class is identical to the one shown in Example 2.

With the Custom Attribute, UsefulMethodMessageAttribute defined I can now go ahead and use it as in the code below.

using System;
namespace MethodLevelAttributes
{
  class Example
  {
    [UsefulMethodMessage(s: "Some pertinent information about the method")]
    public void Foo()
    {
      Console.WriteLine("Calling Foo");
    }
  }
}

Here you can see that the Custom Attribute has been applied to the method Foo.

In the final code example I use reflection to obtain information about the attribute.

using System;
using System.Reflection;

namespace MethodLevelAttributes
{
  class Program
  {
    static void Main(string[] args)
    {
      GetAttribute();
    }

    public static void GetAttribute()
    {
      foreach(MethodInfo mi in typeof(Example).GetMethods())
      {
        UsefulMethodMessageAttribute theAttribute = (UsefulMethodMessageAttribute)Attribute.GetCustomAttribute(mi, typeof(UsefulMethodMessageAttribute));

        if (theAttribute != null)
        {
          Console.WriteLine($"The attribute value is: {theAttribute.S}");
        }
      }
    }
  }
}

Running this you would see the following in the console:

The attribute value is: Some pertinent information about the method

The code for this example can be found here.

Acknowledgements

Eric Lippert 

C# 6.0 and the .NET 4.6 Framework

Writing Custom Attributes

Displaying records from MongoDB in an ASP.NET MVC Application

In a previous post I explained how to save a Twitter stream to a MongoDB using C#. In this post I will show you how to build an ASP.NET MVC application to display those records.

The example that follows was built using Visual Studio 2015 Community Edition and MongoDB 3.4.6.

Firstly this is how the data looks in Mongo. I start a new start Mongo shell, select the database of interest, in this case twitterstream and then using db.auth I authenticate myself as the demouser

and execute a find which displays the data in a hard to read JSON format.  (redacted)

The next step is switch to Visual Studio and create a new ASP.NET Web Application (.NET Framework)

Select Ok.

Accept the defaults and again select Ok.

Now add the following packages via NuGet

  • mongocsharpdriver (Version used 2.4.4)
  • MongoDB.Driver (Version used 2.4.4)

Open the Web.config files and add the following MongoDB entries within the appSettings element.

&lt;add key="MongoDatabaseName" value="twitterstream" /&gt;;
&lt;add key="MongoUsername" value="demouser" /&gt;;
&lt;add key="MongoPassword" value="abcd" /&gt;;
&lt;add key="MongoPort" value="27017" /&gt;;
&lt;add key="MongoHost" value="localhost" /&gt;;

These values identify your MongoDB so if you are using a different database, user etc. modify these values as appropriate. Here are these same values within Web.config

 

 

 

 

 

 

 

 

 

Within the App_Start folder create a new class called MongoContext and add the following code:

using System;
using MongoDB.Driver;

namespace DisplayMongoData99.App_Start
{
  public class MongoContext
  {
    MongoClient _client;
    MongoServer _server;

    public MongoDatabase _database;

    public MongoContext()
    {
      // reading creditials from web.config file
     var MongoDatabaseName = System.Configuration.ConfigurationManager.AppSettings["MongoDatabaseName"];
     var MongoUsername = System.Configuration.ConfigurationManager.AppSettings["MongoUsername"];
     var MongoPassword = System.Configuration.ConfigurationManager.AppSettings["MongoPassword"];
     var MongoPort = System.Configuration.ConfigurationManager.AppSettings["MongoPort"];
     var MongoHost = System.Configuration.ConfigurationManager.AppSettings["MongoHost"];

     // creating creditials
     var credential = MongoCredential.CreateMongoCRCredential(MongoDatabaseName, MongoUsername, MongoPassword);

     // creating MongoClientSettings
     var settings = new MongoClientSettings
     {
       Credentials = new[] { credential },
       Server = new MongoServerAddress(MongoHost, Convert.ToInt32(MongoPort))
     };

     _client = new MongoClient(settings);
     _server = _client.GetServer();
     _database = _server.GetDatabase(MongoDatabaseName);
   }
  }
}

This class setups a connection to the MongoDB instance.  It reads the values from the parameters within Web.config and then calls the MongoClientSettings constructor, setting the  Credentials and Server properties.

Within the Models folder create a new class called TweetModel.cs and add the following code

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;

namespace DisplayingMongoDB4.Models
{
  public class TweetModel
  {
    [BsonId]
    public ObjectId Id { get; set; }

    [BsonElement("thetweet")]
    public string Tweet { get; set; }
  }
}

The TweetModel class represents the data that will be returned by MongoDB. In this case the object id and the tweet.

The next step is to create a new controller. So right click on the Controllers Folder and select Add then Controller. You should then see the following dialog window:

Choose “MVC 5 Controller with read/write actions”, select Add and give it a meaningful name; in this example I have used TweetController. Click add again and Visual Studio will then create the TweetController.cs.

Now amend TweetController.cs as shown below:

using System.Linq;
using System.Web.Mvc;
using DisplayMongoData99.App_Start;
using DisplayMongoData99.Models;

namespace DisplayMongoData99.Controllers
{
  public class TweetController : Controller
  {
    MongoContext _dbContext;

    public TweetController()
    {
      _dbContext = new MongoContext();
    }

    // GET: Tweet
    public ActionResult Index()
    {
      var tweetDetails = _dbContext._database.GetCollection&lt;TweetModel&gt;("tweets").FindAll().ToList();
      return View(tweetDetails);
    }
...
rest of existing scaffolding code here which remains unchanged... 
public ActionResult Details(int id)
...

The changes introduce a new constructor which creates a new MongoContext object. The changes to the Index() method queries the MongoDB database for a collection called “tweets” which will be returned to the View. All the rest of the scaffolding code created by Visual Studio is unchanged.

The final step is right click inside the Index() method and select AddView. This will bring up another dialog window which you should complete as follows:

 

Pressing add results in Visual Studio creating more scaffolding code which when complete should display the following:

 Pressing the “Microsoft Edge” (Or whichever browser you have previously selected) button will launch the Application and after a few moments you will see your MongoDB data displayed.

 

Summary

In this post I have demonstrated how to get started in displaying the content of a MongoDB database from an ASP.net MVC application.

I have skipped over the challenges I encountered in getting the security of my MongoDB set up correctly and that the out of the box display of the data is dreadful at best so if you would like to see further posts on the security or getting the display more production like sound off in the comments.

Acknowledgements

This post would not have been possible without the generosity of these following two posts:

  1. http://www.c-sharpcorner.com/article/simple-crud-operation-using-asp-net-mvc-and-mongodb/
  2. https://www.claudiokuenzler.com/blog/553/authentication-mongodb-3.x-failed-with-mechanism-mongodb-cr

 

Saving a Twitter stream to MongoDB using C#

In my earlier post I built an example of saving a public Twitter stream into RavenDB. Working with RavenDB has piqued my interest with NoSQL databases so in this post I swap out RavenDB and instead use MongoDB to save some of the tweets that appear in the Twitter Public stream

This example was built using Visual Studio Community 2015 and Mongodb 3.4.6

Start MongoDB (on Windows this is mongod.exe ) then start up an Mongo interactive shell (mongo.exe on Windows).  There is no need to create new database, it will be created by the first insert (As a database developer this took some getting use to!)

Start Visual Studio and create a new Console Application. Using NuGet add Tweetini and MongoDBDriver and then type or copy and paste the following, ensuring that you enter your Twitter API credentials on line 21:

using System;
using Tweetinvi;
using MongoDB.Bson;
using MongoDB.Driver;

namespace SavingTwitterStreamToMongo
{
  class Program
  {
    static void Main(string[] args)
    {
      var connectionString = "mongodb://localhost:27017";

      var client = new MongoClient(connectionString);

      IMongoDatabase db = client.GetDatabase("twitterstream");

      IMongoCollection<BsonDocument> collection = db.GetCollection<BsonDocument>("tweets");
      
      // You need to enter your twitter credentials here
      Auth.SetUserCredentials("", "", "", "");

      var stream = Stream.CreateSampleStream();

      stream.TweetReceived += (sender, theTweet) =>
      {
        Console.WriteLine(theTweet.Tweet.FullText);

        var document = new BsonDocument
        {
          { "the_tweet", theTweet.Tweet.FullText }
        };

       collection.InsertOneAsync(document);
     };

     stream.StartStream();
    
     }
   }
 }

This code is the same as shown in this post the only changes are that the RavenDB constructs have been replaced by those required to save the Twitter steam to MongoDB.

Lines 12 and 14 sets up and connects to MongoDB

Line 16 sets the MongoDB database that will be used, in this example it is called twitterstream. (As mentioned earlier in the post, if this database does not exist it will be created on the first insert)

Line 18 shows shows that the tweets are going to be saved inside a collection called tweets.

Line 29 a BSON document is created containing the tweet.

Line 34 saves this document to the database.

Once this code is running, you can switch to the MongoDB interactive shell and review what is being saved.

 

 

 

 

 

 

 

Running the command show dbs you will see the twitterstream database now exists.

 

 

 

 

 

 

 

 

 

 

Switch to the twitterstream database by typing: use twitterstream

 

 

 

Execute the command db.tweets.findOne() and you will see a saved Tweet.

 

 

Saving a Twitter stream to RavenDB database using C#

In an earlier post  I explained how you can use C# to access a Twitter stream. In this post I will show you how to save the tweets from a Twitter stream to RavenDB.

The goal of this post is not to perform a deep dive into NOSQL databases or the Tweetinvi API. Instead its to get you up and running with the minimum of ceremony so you can start conducting your own experiments.

Raven DB is an open source NOSQL database for.NET which as my first experience of a NOSQL database I have found relatively straightforward to start experimenting with.

You can download RavenDB from here.  At the time of writing the stable release was 3.5.3 and I chose to use the installer which then proceeded to install RavenDB via the familiar wizard installation process.

 

 

 

 

 

 

 

 

 

 

 

Once installed you should have a folder structure similar to this:

If, like me you are new the world of NoSQL databases it is worth working your way through the Fundamentals tutorial. I found this an excellent introduction which I highly recommend.

To start RavenDB double click on the Start.cmd batch file in the root of the RavenDB directory. You should shortly see a new command window and a new tab of your default browser showing what databases you have. (which will be empty for the first time launch)

With RavenDB installed and running we can now start Visual Studio and create a new console application. I’ve called mine TrendingOnTwitterNoSQL

Using NuGet, add the following packages:

TweetinviAPI

RavenDB.Client

 

Navigate to Program.cs and add the following using statements:

using System;

using Raven.Client.Document;

using Tweetinvi;

Within the Main method add the following:

Auth.SetUserCredentials("CONSUMER_KEY", "CONSUMER_SECRET", "ACCESS_TOKEN", "ACCESS_TOKEN_SECRET");

Replace COMSUMER_KEY etc. with your Twitter API credentials. If you don’t yet have them. You can obtain them by going here and following the instructions.

Now add the following two lines:

  var stream = Stream.CreateFilteredStream();
  stream.AddTrack("CanadianGP");

The first line creates a filtered Twitter stream. A Twitter stream gives you the developer access to live information on Twitter. There are a number of different streams available. In this post we will be using one that returns information about a trending topic. More information about Twitter streams can be found in the Twitter docs and the TweetInvi docs.

At the time of writing, the Canadian Grand Prix was trending on Twitter which you can see in the second line.

The next step is to create a new class which will manage the  RavenDB document store.  Here is the complete code.

using System; 
using Raven.Client; 
using Raven.Client.Document; 

namespace TrendingOnTwitterNoSQL 
{ 
  class DocumentStoreHolder 
    { 
      private static readonly Lazy<IDocumentStore> LazyStore = 
          new Lazy<IDocumentStore>(() => 
          { 
            var store = new DocumentStore 
            { 
              Url = "http://localhost:8080", 
              DefaultDatabase = "CanadianGP" 
            }; 
            return store.Initialize(); 
           }); 
    
    public static IDocumentStore Store => LazyStore.Value; 
  } 
}

In the context of RavenDB, the Document Store holds the RavenDB URL, the default database etc. More information can be found about the Document Store in the tutorial.

According to the documentation for typical applications you normally need one document store hence the reason why the DocumentStoreHolder class is a Singleton.

The important thing to note in this class is the database URL and the name of the Default Database, CanadianGP. This is the name of the database that will store Tweets about the CanadianGP.

Returning to Program.cs add the following underneath stream.AddTrack to obtain a new document store:

  var documentStore = DocumentStoreHolder.Store;

The final class that needs to be created is called TwitterModel and is shown below

namespace TrendingOnTwitterNoSQL
{
  class TwitterModel
  {
    public long Id { get; set; }
    public string Tweet { get; set; }
  }
}

This class is will be used to save the Tweet information that the program is interested in, the Twitter ID and the Tweet.  The is a lot of other information that is available, but for the sake of brevity this example is only interested in the id and the tweet.

With this class created the final part of the code is shown below

using (BulkInsertOperation bulkInsert = documentStore.BulkInsert())
{
  stream.MatchingTweetReceived += (sender, theTweet) =>
  {
    Console.WriteLine(theTweet.Tweet.FullText);
    var tm = new TwitterModel
    {
      Id = theTweet.Tweet.Id,
      Tweet = theTweet.Tweet.FullText
    };

    bulkInsert.Store(tm);
  };
stream.StartStreamMatchingAllConditions();
}

As the tweets will be arriving in clusters, the RavenDB BulkInsert method is used. You can see this at line 1.

Once a matching Tweet is found, line 3, it is output to the console. Next a new TwitterModel object is created and its fields are assigned the Tweet Id and the Tweet Text. This object is then saved to the database.

The complete Program.cs should now look like:

using System;
using Raven.Client.Document;
using Tweetinvi;

namespace TrendingOnTwitterNoSQL
{
  class Program
  {
    static void Main(string[] args)
    {
      Auth.SetUserCredentials("CONSUMER_KEY", "CONSUMER_SECRET", "ACCESS_TOKEN", "ACCESS_TOKEN_SECRET");

      var stream = Stream.CreateFilteredStream();
      stream.AddTrack("CanadianGP");

      var documentStore = DocumentStoreHolder.Store;

      using (BulkInsertOperation bulkInsert = documentStore.BulkInsert())
      {
        stream.MatchingTweetReceived += (sender, theTweet) =>
        {
          Console.WriteLine(theTweet.Tweet.FullText);

          var tm = new TwitterModel
          {
            Id = theTweet.Tweet.Id,
            Tweet = theTweet.Tweet.FullText
          };

          bulkInsert.Store(tm);

       };
       stream.StartStreamMatchingAllConditions();
     }
   }
 }
}

After running this program for a short while you will have a number of Tweets saved. To view them, switch back to your browser, if not already on the RavenDB page navigate to http://localhost:8080 and click on the database that you created.

 

 

 

 

 

Selecting the relevant database you will then see the tweets.

 

 

 

 

 

 

 

Summary

In this post I have detailed the steps required to save a Twitter Stream of a topic of interest to a RavenDB.

A complete example is available on github

Acknowledgements

The genesis of this post came from the generous answers given to my question on StackOverflow.

Boxing and Unboxing in C#

This post is an aide-memoire as I learn more about boxing and unboxing in C# and is based upon this part of the C# docs.

Boxing

Is the process of converting a value type (such as int or bool) to the type Object or to any interface type implemented by this value type. When the CLR boxes a value type, it wraps the value inside a System.Object and stores it in the managed heap.

Boxing is implicit.

int i = 10; 
// this line boxes i
object o = i;

Although it is possible to perform the boxing explicitly it is not required.

int i = 10;
// explicit boxing
object o = (object)i;

Unboxing

Extracts the value type from the object.

Unboxing is explicit.

int i = 10; 
// boxes i
object o = i;
// unboxes the object to the int value type named j
int j = (int)o;

Performance

Both boxing and Unboxing are computationally expensive operations.

 

Streaming Twitter with C#

In this article I will walk through the steps required to create a C# console application that prints a Twitter stream to the console using TweetInvi library

The example was built using Visual Studio 2015 Community Edition and .NET Framework 4.6.

Step 1

Start Visual Studio and create a new console application, I’ve called mine TwitterPublicStream.

 Step 2

Right click on the project in the solution explorer window (In the example below this is TwitterPublicStream) and select Manage Nuget Packages

 

 

 

 

 Step 3

Search for tweetinvi and once found, install it, accepting the various licences, if you are happy to do so.

 Step 4

In order to use the Twitter API’s, you first need to obtain some credentials. To do this you now need to visit the Twitter API home page and follow the instructions.

Step 5

After that 4 step ceremony we are now ready to write some code.

using System;
using Tweetinvi;

namespace TwitterPublicStream
{
 class Program
 {
 static void Main(string[] args)
 {
 // add your Twitter API credentials here 
 Auth.SetUserCredentials("CONSUMER_KEY", "CONSUMER_SECRET", "ACCESS_TOKEN", "ACCESS_TOKEN_SECRET");

 var stream = Stream.CreateFilteredStream();
 // change LEITOT to something that is currently trending on twitter 
 stream.AddTrack("LEITOT");
 stream.MatchingTweetReceived += (sender, theTweet) =&gt;
 {
   
    Console.WriteLine($"A tweet containing LEITOT has been found; the tweet is {theTweet.Tweet}");
 
 };
 stream.StartStreamMatchingAllConditions();

 }
 }
}

At line 2 a using statement is added for the Tweetinvi library.

At line 11 you need to add your Twitter API credentials that you obtained in Step 4.

At line 15 the AddTrack method is called. Track in the Twitter API context is a comma-separated list of phrases which will be used to determine what Tweets will be delivered on the stream. You can find out more here. Whilst testing this I suggest selecting a trending topic without the #. The one shown in the code was a football game between Leicester and Spurs.

At line 16 the MatchingTweetReceived event will output the contents of the tweet to the console.

Line 21 starts the streaming.

Step 6

In the final step, compile and run the program. After a few seconds you should start seeing Tweets populate the console window.

 

Summary

In this article I have explained how to use the superb library TweetInvi to stream Tweets of interest from Twitter into a C# console application.

C# Utility to emulate the XPath 3 function path()

I recently needed to examine a number of XML files and print out the element names that contained text greater than X number of characters. In addition I also need to print the location of that element within the XML document.

i.e. given the following document

<bookshop>
  <book>
    <title>Microsoft Visual C# Step by Step</title>
  </book>
</bookshop>

…if I was interested in book titles that had more then 10 characters I would want to see:

/bookshop/book/title/
Microsoft Visual C# Step by Step

Whilst it is straightforward to return the text node, finding the XPATH location proved to be more challenging than I initially thought. The reason being is that whilst XPATH 3.0 introduced the path() function that returns the current XPATH location, the number of programming languages that I know (PL/SQL, Python and C#) do not implement XPATH 3.0 yet.

As a result I had to build my own utility. I chose to write this in C# as this is a language I have spent the past 18 months learning and I am now looking for real world problems I can solve using it.

The utility can be found on github. The “engine” of the utility is copied from this Stackoverflow answer: http://stackoverflow.com/a/241291/55640 provided by Jon Skeet.

Although far from feature complete I hope it will give someone facing a similar challenge a head start.

Let me know what you think.