Tuesday, May 26, 2015

Azure Websites now Web Apps

I have finally been able to get into using Windows Azure more and I'm loving it. I had been using it for the last two years on and off, but with the recent updates announced at Build, it's been amazing.

I found some great articles from Rick Anderson at Microsoft about Identity, MVC5, Azure, and Identity (including OAuth2.0) The series found here gets you to a production worthy site in just a few hours that includes user registration, SMS, external logins, and all that base framework that just about any site needs.

So, here's the scenario...

I started a project in MVC4 and would like to migrate to MVC5 and use the ASP.NET Identity stuff with Entity Framework and Azure SQL.

I haven't found an article describing how to use an existing database with an existing project that did not originally use that data base. For example, the MVC4 site used abc.sqlserver.qualifiedname. After updating, I'd like to use def.sqlserver.qualifiedname on a BLANK database.

Using EF migrations enables this scenario.

From the package manager, use the following commands:

PM> enable-migrations
PM> add-migration Initial

Update your connection string for the connection which points to the users db (eg, DefaultConnection) from abc (original) to def (new azure sql) server and set initial catalog to the blank database.

PM> update-database

the first command adds the required components to enable migrations including adding a Migrations folder to you project.

second command creates a migration representing the current DB schema

third command actually sends commands to the azure sql server that you updated in the web config above to be like the original database.

Migrations are a great way of helping to enable continuous deployment of databases so I'm hoping EF turns out to be as great as everyone says :)

Friday, April 24, 2015

It's official - I can program in C#

I finally went and took the first of three exams required to become a Microsoft Certified Software Developer (MCSD).  I opted for the 70-483 Programming in C# exam over the 70-480 Programming in HTML5 with JavaScript and CSS3, though I do plan to take that in the future.

Tips:
  1. Study specifically for the exam
    The range of topics is quite large and it is likely there will be questions you are not familiar with.  Topics I was not familiar with:
    • Performance Counters
    • GAC tools
    • Specific Crypto Algorithms
  2. Carefully read each question
    There can be subtle verbage that helps immediately rule out answers.
    Eg, You need to send large volumes of data securely using a hashing algorithm.  
    This would rule out any encryption algorithms.
  3. Get comfortable with the query syntax of LINQ
    I never use the 'query syntax' (eg, from c in collection where ...) and just about every question involving LINQ used that form
Overall
The exam was 55 questions with a maximum allowed time of 120 minutes.  This was more than enough time to read and think about each question before answering.  I finished the exam in just under an hour and never felt time pressure.  At the end, I had mistakenly not answered one question and it informed me of this before letting me end the exam so don't worry about skipping questions.  I do recommend taking the time to read and understand each question which I can't stress enough.  Several times, the supplied code in the question gave away the answer, it just took a second to realize what was being asked.

Bonus
I didn't realize going into it, but by passing a single exam, you are considered a Microsoft Certified Professional (MCP) and have special recognition in the Microsoft Community. I'm excited to start digging into what's included. More on this to come :)

Friday, March 27, 2015

MVC and MVVM Together in Modern Web

I've spent the last few years heavily in the web space and really believe that we're finally in a pretty good spot with design patterns, support from browsers, web standards, client hardware (smart phones) that can do more and more complex things...  But there is still so much going on and things are changing quickly!

I want to look at how to use the MVC and MVVM patterns together to provide a very powerful client application.

Lets start by breaking down these patterns.

Model

Models are typically our business objects.  More specifically, these should be Business Value Objects.  I like to make that distinction as these objects should be purely data (no methods).  How are you going to transmit the method to the client with your model?

View

A representation of data.  A view uses a model to construct some representation, typically a visual representation.  The view is in charge of binding the model to the eventual markup in the case of ASP.NET MVC.

Controller*

A controller is something that handles requests and return values.  I am leaving this slightly vague at the moment, but the main point is that they handle requests.

View Model

Typically a composition of models and other view models that are able to manipulate client side objects, and manage user interaction to the system.  For example, if you have a button on a page, there should exist some view model that will handle the click event.  Typically, this event handler will call out to a controller, take a response asynchronously, and update the view accordingly.

That leads us to the second reason for a view model - to control the state of the page.  Whether an expander is opened or closed should be driven by an observable property of the View Model.

The View Model is delivered to the user in the View returned by the Controller after a request.


Let us consider a simple web example.
  1. User makes a request to my site to see a product catalog
  2. Request for markup is handled by a Controller
  3. Controller Finds a View based on the Route of the request
  4. Controller Finds a Model based on the arguments of the request (route and/or message body)
  5. The Model is given to the View to construct some visual representation of the Model (render HTML)
  6. Within the generated markup, a view model must exist to handle user interaction and manage page state.  This is typically going to be some JavaScript object
  7. Markup is delivered to the client and the View Model is initialized
  8. Once the view model is initialized, it may call back to the server to fetch some data (eg, product list meta data)
  9. Request for data is handled by a Controller
  10. Controller does not need to find a view since we are requesting data, it simply finds the Model and returns it as data
  11. The View Model receives the data asynchronously and updates the markup by rendering the list of products via template binding
At this point our user experience is improved as the site is loaded quickly to deliver the layout, minimal data, but a view model that can take over.  This improved responsiveness is very noticeable.  Consider the alternative where the all the data must be fetched before the view can be rendered and delivered to the client.  The client gets a blank web page with a spinner.

So, you can see, there are really two types of requests we are dealing with here.  Controllers that respond to requests with markup are typically referred to as MvcControllers.  Controllers that respond to requests with data (json, xml, file, ...) are typically referred to as ApiControllers.

In the above example, we're using MVC to get us into some interesting section of the application.  After the View is delivered, MVVM takes over and the View Model handles user interaction and manages page state.  Using MvcControllers to deliver data is not natural or easy if you are using ASP.NET MVC.  WebAPI (aka, ApiControllers) were developed specifically to provide an easy way to provide data formatted in a commonly understood format (json, xml, ical, ... think mime).

I find this is a very powerful approach where the View Models become unit testable instead of having to be tested via CodedUI or similar.  Once the View Model is tested, it's just a matter of binding a view which isolates errors to binding errors.

Overall, I think this is going to become one of the more common patters to see when developing new, modern web applications.

WCF Best Practices Part 1: Setting up the solution

Windows Communication Foundation (WCF) is one of the most commonly used technologies to connect to services in the Microsoft Stack.  I found it surprisingly hard to find any guidance on how to setup your projects when using WCF.  Please refer to this dnrtv episode for guidance on how to setup your projects.  It's an oldy, but still applies today.

Some of the biggest take aways that I finally uncovered from this episode were:
  1. Service References are not your friend - DO NOT USE THEM!
  2. There are 5 major components when dealing with WCF
While I noted that service references are not your friend is the number 1 take away, I will cover the 5 components first as that will bring some context as to why they are bad.

1.  Contracts

Contracts are the core foundation of any service and are the most critical aspect to get right as early as possible.  The contracts describe the agreement that a client has with the service code driving it.  Contracts are not secret, in fact, that would be counter productive.  I want to consume a service, but I do not know how to interact with that service.  How would I know how to request something, to even how to talk to it (eg, TCP, HTTP, ...)?

In the contracts, you will have two primary types of objects:
  1. Data Transfer Objects - DTOs
  2. Service Interfaces
Notice, only the interface to the service exists in this project, not the implementation code.

Contracts will exist in their own project so that they can be shared with clients (you can openly distribute this to your customers).  If the customer is able to use your contracts dll directly, less work for them.  If the customer cannot use your contracts dll directly, WCF supports discovery while allows tools to auto-generate types (eg, Service References).

I highly, highly recommend decorating all of your DTOs with the [DataContract(NameSpace="something.that.is.not.the.default")]

This project is typically a Class Library

2.  Client Proxy

This project contains proxy classes implementing the service in question.  In .net, these classes would look something like:

// in the Contracts project
public interface IDoStuff
{
    ReturnedStuff DoStuff(WithStuff thisStuff);
}

// in the client Proxy
public class IDoStuffProxy : ClientBase<IDoStuff>, IDoStuff
{
    ReturnedStuff DoStuff(WithStuff thisStuff)
    {
        return this.Channel.DoStuff(thisStuff);
    }
}

This can easily be auto-generated and is not secret so you can provide it openly to customers.  Tools like Service Reference will generate these proxy types for you.  As you can see, there is not much involved with generated these by hand even.  svcutil (which Service References uses) can generate these for you as well.

This project is typically a Class Library

3.  Service Implementation

Of course you'll eventually have to provide some working implementation for the service contracts in the contracts project.  This type will also contain additional types that help support the service, but which are not simply DTOs or should not be given out to the customer because they contain some proprietary functionality, etc.

This is your secret sauce and should not be given out.  This code can be updated at any time as long as it does not break the contract with the client.  We know it will not as that would require a change in the Contracts project.

This project is typically a WCF Service Library

It is imperative that this be host independent as you may want to have on prem, azure, aws, ... or even iis and apache.  Trust me, you are better off keeping this separate from the service host.

Service Libraries also have a nice feature where the WcfTest client will open with a reference to your service if you start the service library directly.

4.  Service Host

If you are using iis and/or iis express the service host is extremely simple.  It contains only the web.config file and transforms.  That's it.

This project is typically a WCF Service Application

5.  Client

Finally, we have the client.  A client is anything that uses your service.

Why 5 Separate Projects

Primarily, it keeps you honest.  You should not change an API to suite the needs of an implementation and when the interface the service implements resides in a completely separate project, it really makes you consider if that is the right thing to change.

It has the added benefit of providing bits you can provide to customers which are not secret so they don't have to generate those types.  Think, Barrier to Entry.

Finally, it lends itself to easy deployment to multiple environments and even multiple platforms.

Service References

Are not your friend because Visual studio tries to manage things for you and when things break, you have no idea what you are looking at; worse, it sometimes breaks more when you 'fix' it.  Specifically, it tries to generate the DTO and proxy types and manage the web config file related to those services (it is very, very bad at this management).  Under the covers, it uses the svcutil.exe which you can use as well from command line and I guarantee you will do a better job!

The only reason to use the Service Reference would be to have it generate the types which you then cleanse and add to source control as your own code, the remove the service reference.  At that point, you may as well use the svcutil.exe.

When you own the contracts dll (maybe another team within the company) or the service provider just gives out the contracts dll directly, use it*.

*One nice thing the svcutil.exe can do is generate Task-based methods of the service interface, even when the service interface is not Task-based.  In this case, you may want to generate your own types anyway.


This was the biggest missing link for me when I jumped into WCF.  I started after version 4.5 was released so the configuration files were super easy to figure out.  I just didn't understand how to structure the projects so hopefully someone out there benefits from this!  :)

Tracing is EVIL: Part 1

So, for the last year or so I've been looking into application logging and event tracing.  I looked into ETW after listening to a .net rocks episode a long time ago, but quickly stopped when I realized that starting and stopping the tracing required elevated privileges on the server.  I haven't worked at many places where that would even be up for disucssion...  So I eventually came across System.Diagnostics.TraceSource.  This seemed to be a good fit as I could turn it on and off via configuration file and I could add custom listeners to defer the 'write' to be handled by some external handler(s).

The application under test is built around interfaces and base classes implementing the tracing where constructed.

Let's bring in an example...

/// interface for loading stuff
public interface ILoadStuff {
    /// load the stuff with the specified id
    Stuff LoadStuff(int id);
}

public class TracingStuffLoaderWrapper : ILoadStuff {
    protected readonly ILoadStuff wrapped;
    protected readonly TraceSource trace;
    public TracingStuffLoader() : this(new StuffLoaderProxy()/*WCF/service proxy*/, new TraceSource("NS.TracingStuffLoaderWrapper", Information)){}

    public TracingStuffLoader(ILoadStuff wrapped, TraceSource trace){
        // arg null check
        this.wrapped = wrapped;
        this.trace = trace;
    }

    public Stuff LoadStuff(int id){
       trace.TraceInformation("Request to load stuff with id `{0}`.", id);
       var stuff = wrapped.LoadStuff(id);
       trace.TraceInformation("Stuff fetched successfully");
       return stuff;
    }
}

or

public abstract TracingStuffLoaderSql : ILoadStuff {
    protected readonly TraceSource trace;
    protected TracingStuffLoaderSql() : this(new TraceSource("NS.StuffLoaderSql")){}
    protected TracingStuffLoaderSql(TraceSource source){
        this.trace = source;
    }

    protected abstract SqlCommand GenerateLoadCommand(int id, SqlCommand cmd);
    protected abstract Stuff DecodeStuffFromDataSet(DataSet ds);
    public Stuff LoadStuff(int id){
        trace.TraceInformation("Request to load stuff with id `{0}`.", id);
        using(SqlConnection conn = GetOpenConnection()){
            trace.TraceInformation("Generating load command.");
            using(SqlCommand cmd = GenerateLoadCommand(id, conn)){
                trace.TraceInformation("Fetching data from database.");

                DataSet ds = new DataSet();
                new SqlDataAdapter().Fill(ds);
                trace.TraceInformation("Dataset fetched with {0} tables and {1} rows in the first table.", ds.Tables.Count(), ds.Tables.First().Rows.Count());

                Stuff decoded = DecodeStuffFromDataSet(ds);
                trace.TraceInformation("Decoding succeeded: {0}", decoded);
                return decoded;
            }
        }
    }
}


Using this pattern, the StuffLoader can work against any Sql database, just implement the SqlCommand method in the derived class.  The subclass would hopefully do the right thing and use the trace source to write messages, but even without writing to it, this gives a very good idea as to where a failure may have occurred..

If the trace prints "Generating load command." and does not print "Fetching data from database." the failure is inside the GenerateLoadCommand subclass implementation.  Otherwise, my base class is broken and I should have a good idea where.

This approach was working fine as there were only a few traces happening here and there.  After introducing tracing into just about every operation in the system I've uncovered a drastic flaw in this approach!!!

This SO post shares exactly my feels.  I could not believe that the tracing was synchronous and that the default trace listener was so horribly performing.  The app actually started hanging and crashing from unresponsive tracing.  I insisted that it could not be the tracing as it *must* be asynchronous....

When profiling the application locally, except for start up, the app has very little resource use (<5% CPU, < 100MB RAM) across 3 WCF services and a website.  With as few as 10 users, the site went from blazing fast to painfully slow.

Lesson learned.... DO NOT USE THE DEFAULT TRACE.  However, no matter what I tried, I could not get the default trace to stop outputting.  I had to disable tracing completely and enable it only after an error is observed; try to reproduce to capture (while horribly impacting the running application for all other users), and disable.  This really sux.

Anyway, so I've known about ETW for a while, but haven't really tried to use it.  I feel like ETW is the best choice for what I'm striving for.  In fact, my misunderstanding of how TraceSource works is exactly how EventSource works, so lucky me :)

So, how do you convert between old TraceSource (.net 2.0 feature) stuff and implement the new EventSource stuff (.net 4.5/.1 feature), you ask?

Stay tuned for Part 2!









Thursday, March 5, 2015

WCF + WebAPI + Identity space

It's been quite a while since I've been able to finish a blog article (I still have quite a few pending), but I really want to get *something* out here just to get back into the habit.

Lately, I've been thinking about Identity and Microservices.  Today, there does not seem to be a great way to share identity from a client to a server, then to another (or many other) services.

I thought federated identity would provide a better mechanism, but it still just provides a single authentication point for the user; it doesn't really handle passing a request between services.

I recently listened to a .net rocks post about Identity Server and saw an ndc video about it.  There are a bunch of videos on ndc around identity and I'm excited to see the concept of an identity server for an application domain (all the microservices that comprise the actual service).

I'll have to investigate this further and see just how well OAuth and OpenID Connect really work.

Along those lines, I've been hearing more and more about WCF being 'dead' and WebAPI being the new standard.  Unfortunately, when I first heard about WebAPI years ago, I thought the same thing, but I didn't really understand why you would pick one over the other.  Having spent the last few years in this space pretty heavily, I've noticed a few things...

Firstly, I always hear about how hard WCF is to configure.  However, that has not been my experience.  I have had nothing but an awesome time using WCF.  I think the fact that I came in at version 4.5 has to do with my rosy attitude toward WCF.

File-less activation of services and default binding behaviors makes deploying new services extremely easy.

WebAPI has been a blast as well.  From my javascript-based web application, I can call into my services with ease.  A simple get or post as the method type, a simple attribute on my api controller and we're good; my service has an object and my client has HTTP.

I think one of the major road blocks to getting more posts published is finding a way to show code and discuss it via text.  I will be trying a youtube channel to discuss my thoughts about various technologies and link them from my blog with any supporting content.

Stay tuned for a short presentation discussing when to use WCF and when to use WebAPI.

Wednesday, October 29, 2014

CLR Types in SQL Server - Investigation Only

Today, I'm finally digging into a topic that I've wanted to look at for a while:  Using CLR Types in SQL Server.  I've been interested in this topic for a while now; especially when I see very complicated stored procedures implementing business logic that would be simple in C#.  This may very well be that the teams I've worked with simply do not know the 'proper' way of writing these procedures and therefore cannot achieve optimal performance and/or maintainability / readability.

So, I'm a developer that is very comfortable in C#, but moderately comfortable with SQL, but mainly from an analytics stand point.  Denormalizing tables and building cubes is one thing, creating a highly performant transactional system is definitely another.  So any example SQL code that I may use for comparison is coming from a software developer's mind, not a SQL Server administrator's mind.

Here's the scenario:

As a developer primarily focused in C# you are tasked with refactoring a rules engine built in SQL Server using Stored Procedures filled with business logic and tables to provide rules used by the Stored Procedures.

My reaction: 

I'm going to mess up this SQL, but I know I can do it in C# with unit tests!

However, being the practical guy that I am, I first turned to the community for their opinion.  Overall, it seems the community favors business logic in an application / business logic middle-tier.  I am included in that group, however, I'll try to consider the alternative and it's merits.

Having stored procedures in SQL Server allows the processing of the logic to be close to the data.  There is no client application as we are effectively ETL-ing data from source to the Data Warehouse so another tier would be required where using procs that tier is unnecessary.  Since it's ETL, there is a lot of scanning over batches or records and data access from T-SQL is going to be faster than other approaches.

CLR routines cover the first two conditions where the logic would be close to the data and there would not be another tier required.  It would introduce an assembly to care for, however.  While T-SQL is going to be faster for data access, processing the rules may be way faster in C# using the CLR.  Typically, we process nearly continuously so the amount of data to process is small.  If we have to reprocess all the historic data, we'll see if the read performance improvement of T-SQL out weighs processing performance of C# using the CLR.

Bottom line is there are 3 options:

1.  Improve my SQL skills to be able to write complex business rules in Stored Procedures.
2.  Write the rules in C# and register an assembly to SQL Server.  This would be able to be marked as SAFE as it simply ETLs from source to DW.
3.  Create an application / business rules layer using C# that processes the data continuously.

#2 seems the most realistic option for me so we'll investigate that now!  :)

From the Docs

As with most Microsoft things that I dive into, I like to start with a deep dive of the info available on MSDN.

Performance

To my very happy surprise, on the first page of the docs
  • Potential for improved performance and scalability. In many situations, the .NET Framework language compilation and execution models deliver improved performance over Transact-SQL.
Now to figure out what those situations are...  As I continued into the section about Performance of CLR Integration,

CLR functions benefit from a quicker invocation path than that of Transact-SQL user-defined functions. Additionally, managed code has a decisive performance advantage over Transact-SQL in terms of procedural code, computation, and string manipulation. CLR functions that are computing-intensive and that do not perform data access are better written in managed code. Transact-SQL functions do, however, perform data access more efficiently than CLR integration.

Ugh, right back where I started.  The function is computing intensive, but requires "lots" of data access.  Again, the typical scenario being just the delta since last processing finished; is that really a lot?  Even reprocessing all historic data... is it a lot?  And are the rules complex enough to notice any performance benefit?

Then, we always have to keep in mind there are other costs aside from performance.  Is the cost of the stored procedure approach (readability / maintainability / team skills) higher than the cost of the CLR approach and does that cost out-weigh any potential performance benefits?

So far, more questions than answers for sure.  At least for now I know it has the potential to be a good solution.

Security

There are a few security mechanisms available for restricting code run by an assembly.

This assembly will only ETL data from a staging table with the delta to Source since the last processing completed so we can use SAFE.  Safe is the most restrictive setting and cannot access resources outside of SQL Server.  You can read about the other options on MSDN using the above link.

When registering the assembly with SQL Server, it will ensure that the assembly does not contain code that would not be allowed to run when called.  This prevents run time errors from occurring due to using external resources.  Luckily, it can detect code that will not be called and will allow the assembly to be loaded even if that un-callable code contains code that would otherwise cause the assembly to error when loading.

SQL Server Types in CLR

The SqlTypes namespace is used to provide the same semantics and precision found in SQL Server.  This is especially important for NULLability when comparing, numeric precision, and streaming LOB types.  The type mapping between CLR and SQL Server can be found here.

I'm interested in the streaming LOB and Streaming Table Valued Functions (STVFs).  I did not realize such a thing existed and I'm wondering if this could help performance as well.

Versioning and User Defined Persisted Types

One of the features I was unaware of is that you can create CLR data objects and save them directly into a column of that type.  There is a SqlUserDefinedType that can be used to describe how to save the object.

Because UDTs are accessed by the system as a whole, their use for complex data types may negatively impact performance. Complex data is generally best modeled using traditional rows and tables. UDTs in SQL Server are well suited to the following:
  • Date, time, currency, and extended numeric types
  • Geospatial applications
  • Encoded or encrypted data
Ok, so what does "accessed by the system as a whole" mean...  Further what is complex, anything with a child object?  Does JSON serialized data count as encoded?

User Defined types must implement the INullable interface in SqlTypes as SQL Server is 'null-aware'.  There are a ton of other restrictions as well.

UDTs can be read from a data reader and cast to the object's type directly.  The client code must, of course, have a reference to the assembly containing the UDT.  The object can also be read as bytes and no reference is required in that case.

SqlParameter can have its type set as SqlDbType.Udt and that type can be passed as input to the procedure.

Overall, my first impression of UDTs is that they are extremely complicated for anything beyond simple data types.

CLR Methods

Custom CLR Attributes

Methods in a CLR routine can be attributed with SqlFunction attribute and specify a DataAccess parameter to help SQL Server understand the method's intent (eg, read data from a table).

CLR vs T-SQL Table Valued Functions

T-SQL TVFs return a table that can have unique indexes and constraints since it is an actual table.  CLR does not do this, but instead has a streaming interface via IEnumerable.  As soon as the first record is available, the calling code can start consuming.

A user-defined table type cannot be passed as a table-valued parameter to, or be returned from, a managed stored procedure or function executing in the SQL Server process.

Can a UDT in CLR be used?

Pass by reference when using OUT parameters.


CLR Stored Procedures

Returning Tabular Results and Messages

Returning tabular results and messages to the client is done through the SqlPipe object, which is obtained by using the Pipe property of the SqlContext class. The SqlPipe object has a Send method. By calling the Send method, you can transmit data through the pipe to the calling application.

Returning Tabular Results

To send the results of a query directly to the client, use one of the overloads of the Execute method on the SqlPipe object. This is the most efficient way to return results to the client, since the data is transferred to the network buffers without being copied into managed memory.

Implementation

Unfortunately, I was unable to refactor the SQL Rules to compare against the CLR routines as the procedures were beyond my ability to comprehend.  I do hope to return to this topic soon, but at least my initial investigation and notes are here for when that day comes! :)