Writing unit tests for an analyzer project

In this section, we will show you how to write and execute unit tests for an analyzer project.

Getting ready

You will need to have created and opened an analyzer project, say CSharpAnalyzers in Visual Studio 2017. Refer to the first recipe in this chapter to create this project.

How to do it...

1. Open UnitTests.cs in the CSharpAnalyzers.Test project in the Solution Explorer solution to view the default unit tests created for the default symbol analyzer (type names should not contain lowercase letters) for the template analyzer project.

2. Navigate to Test | Windows | Test Window to open the Test Explorer window to view the unit tests in the project. Default analyzer project has two unit tests:
   • TestMethod1: This tests the scenario where analyzer diagnostic does not fire on the test code and
   • TestMethod2: This tests the scenario where analyzer diagnostic does fire on the test code.

Note that the unit test project contains unit tests for both the DiagnosticAnalyzer and CodeFixProvider. This chapter deals with analyzer testing only. We will expand on the unit tests for the CodeFixProvider later in this book.

3. Run all the unit tests for the project by right-clicking Not Run tests node in the Test Explorer, executing the Run selected tests context menu command, and verify that the tests pass.

 

4. Edit TestMethod1 so that the test code now has a type with lower-case letters:

[TestMethod]
public void TestMethod1()
{
  var test = @"class Class1 { }";

  VerifyCSharpDiagnostic(test);
}

5. Right-click on TestMethod1 in the editor, execute the Run tests context menu command, and verify that the test now fails with the diagnostic mismatch assert - expected "0" actual "1":

6. Edit TestMethod1 to now add an excepted diagnostic for the new test code:

var expected = new DiagnosticResult
{
  Id = "CSharpAnalyzers",
  Message = String.Format("Type name '{0}' contains lowercase letters", "Class1"),
  Severity = DiagnosticSeverity.Warning,
  Locations = new[] {
    new DiagnosticResultLocation("Test0.cs", 11, 15)
  }
};

VerifyCSharpDiagnostic(test, expected);

7. Run the unit test again and note that the test still fails, but now it fails due to a difference in the location (column number) at which the diagnostic was reported.

8. Edit the diagnostic location to use the correct expected column number and rerun the test - verify that the test passes now.

new DiagnosticResultLocation("Test0.cs", 11, 7)

9. Edit TestMethod1 and change the test code to rename Class1 to CLASS1:

var test = @"class CLASS1 { }";

10. Run the unit test again and verify that the test fails now due to a diagnostic mismatch assert - expected "1" actual "0".

11. Edit TestMethod1 to remove the expected diagnostic and verify the test passes:

 var test = @"class CLASS1 { }";

 VerifyCSharpDiagnostic(test);

How it works...

The analyzer unit test project allows us to write unit tests for the execution of our analyzer on different code samples. Each unit test is marked with a TestMethod attribute and defines sample test code, expected diagnostic(s) reported by the analyzer on that code (if any), and invocation of test helper method(s), here VerifyCSharpDiagnostic, to verify diagnostics.

//No diagnostics expected to show up
[TestMethod]
public void TestMethod1()
{
  var test = @"";

  VerifyCSharpDiagnostic(test);
}

Unit tests can define expected diagnostics using the DiagnosticResult type, which must specify the diagnostic Id, Message, Severity and Locations for the diagnostic:

var expected = new DiagnosticResult
{
  Id = "CSharpAnalyzers",
  Message = String.Format("Type name '{0}' contains lowercase letters", "Class1"),
  Severity = DiagnosticSeverity.Warning,
  Locations = new[] { new DiagnosticResultLocation("Test0.cs", 11, 15) }
};

VerifyCSharpDiagnostic(test, expected);

Computing the correct line number and column number for an expected diagnostic, for example (11, 15), can be a bit tricky. The approach that normally works is to start with a default location of (0, 0), execute the test once, and look at the failure text in the Test Explorer window to get the expected and actual line number. Then, replace the expected line number in the test code with the actual line number. Re-execute the test and repeat the process to get the correct column number.

UnitTest type containing all the unit tests also overrides the following methods to return the DiagnosticAnalyzer (and optionally a CodeFixProvider) to be tested:

 protected override CodeFixProvider GetCSharpCodeFixProvider()
 {
   return new CSharpAnalyzersCodeFixProvider();
 }

 protected override DiagnosticAnalyzer GetCSharpDiagnosticAnalyzer()
 {
   return new CSharpAnalyzersAnalyzer();
 }

Now, let us expand a bit more on the test framework helpers for the unit tests. The analyzer unit test project contains two primary helper abstract types to write unit tests for analyzers and code fixes:

• DiagnosticVerifier: Contains helper methods to run DiagnosticAnalyzer unit tests that verify the analyzer diagnostics for a given set of test sources.
• CodeFixVerifier: Contains helper methods to run DiagnosticAnalyzer and CodeFixProvider unit tests that verify the analyzer diagnostics for a given set of test sources before and after applying a code fix. This type derives from DiagnosticVerifier.

In the default analyzer project, UnitTest type derives from CodeFixVerifier, but could also be changed to derive from DiagnosticVerifier, if you are only interested in writing analyzer unit tests. We will just focus on the DiagnosticVerifier here; CodeFixVerifier is covered later in chapter.

DiagnosticVerifier type is split into 2 source files DiagnosticVerifier.cs and DiagnosticVerifier.Helper.cs.

• DiagnosticVerifier.Helper.cs contains the following core functionality:
  • Helper methods to create a compilation with source files based on given C# or VisualBasic source code (Set up compilation and documents region in the preceding screenshot).
  • Helper methods to invoke the preceding functionality to create a compilation with the given C# or VisualBasic source code, and execute the given DiagnosticAnalyzer on the compilation to produce analyzer diagnostics and return sorted diagnostics for verification (Get Diagnostics region in the preceding screenshot).
• DiagnosticVerifier.cs contains the following core functionality:
  • Method(s) to get the DiagnosticAnalyzer type to be tested (To be implemented by Test classes region in the preceding screenshot).
  • Private helpers to perform actual diagnostic comparison and verification and formatting of diagnostics to get a string representation of actual/expected diagnostics when a unit test fails (Actual comparisons and verifications region and Formatting Diagnostics region in the preceding screenshot).
  • Diagnostic verification methods VerifyCSharpDiagnostic and VerifyBasicDiagnostic that can be invoked by the unit tests to verify analyzer diagnostics generated on the given C# or Visual Basic source code (Verifier wrappers section in the preceding screenshot). These methods invoke the Get Diagnostics helpers to create a compilation and get sorted analyzer diagnostics and then invoke the preceding private helpers to compare and verify diagnostics.

See also

Live Unit testing is a new feature in Visual Studio 2017 Enterprise Edition, which automatically runs the impacted unit tests in the background as you edit code, and visualizes the results and code coverage live, in the editor, in real-time. Refer to Chapter 6, Live Unit Testing in Visual Studio Enterprise, to enable live unit testing for the project and visualize unit tests automatically executing after you edit the code in the steps in this recipe.