Tuesday, September 23, 2008

VSTS Code Coverage Bug?

I am doing code coverage analysis with Visual Studio, which is generally an easy task now with the tools that are included. However, I have an issue that I can't overcome. Let's say I have assemblies A, B, C, and D and have marked them all for coverage analysis. I run the tests and look at the results and find a report that contains A, B, and C - but not D. I investigate and find that no tests actually execute any code in D (let's say it's the asp.net front end and I don't leverage UI testing yet). Because there are no tests for D causing D to be missing from the report the total code coverage percentage and "blocks not covered" are incorrect.

Does anyone know how I can do either of the following?
  • Calculate the total "number of blocks" in D so that I can manually adjust the coverage report to be correct?
  • Get the Coverage report to automatically show the number of blocks not covered for assemblies that are instrumented for coverage but are not tested at all?

While I do want test coverage to improve I am analyzing coverage reports saved at historic points in time in the code base. Thus I don't want to create a test that simply executes at least 1 block of code in each assembly and the re-calculate test coverage by running the tests. That would be a pretty time consuming work-around to something that seems like a simple problem.

Wednesday, September 3, 2008

TeamReview makes the list

TeamReview has made it onto the definitive Team System Widget list!

A generous thank you goes out to Team System MVP, Richard Hundhausen, of Accentient for not only inlcuding TeamReview on the list but for also providing me with a tremendous introduction to Team System in a TFS Deep Dive workshop I attended through Accentient a few years ago. The course material is still at my desk, and I still use it.

Tuesday, September 2, 2008

Using GhostDoc in a macro to comment an entire solution

At work we've started leveraging Visual Studio Team System to essentially automate the checking of Standards and Best Practices (S&BPs). We decided that anything that could be automatically enforced could be deemed part of the S&BPs because it was cost effective. Anything else could be deemed guidance but would not be required. One of the aspects that we desired was being able to effectively use Sandcastle to generate technical documentation. I personally feel that the most valuable documentation from XML comments comes form the comments on Acceptance Tests since they define the expectations of the system. To make those XML comments required we created a shared MSBuild file that treated some of the XML compiler warnings as errors.

Playing Catch-Up
This unfortunately left us with a lot of catch-up work. We needed to fill in and correct thousands of missing comments. While it would have been the most valuable to add thoughtful defined explanations in those places it also would have been the most costly - and it wasn't in our project budget to take on that cost. As a result I adapted a Visual Studio macro with some code-DOM traversing material borrowed from TeamReview in order to leverage GhostDoc to automatically fill in missing comments and fix some comment structure for an entire Visual Studio solution. This allowed us to take on the endeavor of changing our S&BPs, being able to enforce them for the better, and allowed us to move forward without immediately paying for Technical Debt due to the change in S&BPs.


Again, GhostDoc comments are meant to get us over the hurdle of not being able to enforce a coding standard. I know as well as you that auto-generated stuff isn't very valuable, but without it we wouldn't be able to approach this endeavor.

Before you run the macro
Be sure your code compiles and that you have the entire solution checked out from source control. Otherwise you'll get lots and lots of prompts, potentially two for every file.

Below is the code. The macro is named after the Winston Wolf character in the movie Pulp Fiction.

The Code
Option Explicit Off
Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics

Public Module TheWolf

Public Sub CleanUpSolution()
IterateFiles()
End Sub


Private Sub IterateFiles()
Dim project As EnvDTE.Project
Dim projects As EnvDTE.Projects
Dim window As Window
Dim target As Object
DTE.ExecuteCommand("View.
SolutionExplorer")
window = DTE.Windows.Item(Constants.vsWindowKindCommandWindow)
projects = DTE.Solution.Projects
If projects.Count = 0 Then
Exit Sub
End If


For Each project In projects
If (Not project.ProjectItems Is Nothing) Then
CleanUpProject(project.ProjectItems())
End If
Next
'RebuildSolution()
End Sub

Private Sub CleanUpProject(ByVal items)
Dim file As ProjectItem
For Each file In items
If file.Name.EndsWith(".cs") OrElse file.Name.EndsWith(".vb") Then
ActivateFile(file)
RemoveAndSortUsings()
FormatDocument()
DocumentCode(file)
End If

'Handle folders within a project
If Not file.ProjectItems() Is Nothing Then
If file.ProjectItems.Count > 0 Then
CleanUpProject(file.ProjectItems())
End If
End If
SaveAll()
CloseAll()
Next
End Sub

Private Sub RebuildSolution()
DTE.ExecuteCommand("Build.RebuildSolution")
End Sub

Private Sub SaveAll()
Try
DTE.ExecuteCommand("File.SaveAll")
Catch ex As Exception
End Try
End Sub

Private Sub CloseAll()
Try
DTE.ExecuteCommand("File.CloseAllButThis")
DTE.ExecuteCommand("File.Close")
Catch ex As Exception
End Try
End Sub

Private Sub ActivateFile(ByVal file As ProjectItem)
file.Open()
If (Not file.Document Is Nothing) Then
file.Document.Activate()
End If
Try
DTE.ExecuteCommand("View.ViewCode")
Catch ex As Exception
'do nothing - it's probably already viewable
End Try
End Sub

Private Sub RemoveAndSortUsings()
Try
DTE.ExecuteCommand("Edit.RemoveAndSort")
Catch ex As Exception
'does not work on VB code
End Try
End Sub

Private Sub FormatDocument()
DTE.ExecuteCommand("Edit.FormatDocument")
End Sub

Private Sub DocumentCode(ByVal file As ProjectItem)
Dim element As EnvDTE.CodeElement
Dim document As EnvDTE.TextDocument
document = CType(file.Document.Object(""), EnvDTE.TextDocument)
If (document Is Nothing) Then
Return
End If
DocumentElements(document, file.FileCodeModel.CodeElements)
End Sub

Private Sub DocumentElement(ByVal document As EnvDTE.TextDocument, ByVal element As EnvDTE.CodeElement)
Dim startPoint As EnvDTE.EditPoint
document.Selection.GotoLine(element.StartPoint.Line, True)
startPoint = document.CreateEditPoint(document.Selection.ActivePoint)
document.Selection.MoveToPoint(startPoint, True)
DTE.ExecuteCommand("Weigelt.GhostDoc.AddIn.DocumentThis")
End Sub

Private Sub DocumentElements(ByVal document As EnvDTE.TextDocument, ByVal elements As EnvDTE.CodeElements)
Try
For Each element In elements

Select Case element.Kind
Case vsCMElement.vsCMElementFunction
Dim func As EnvDTE.CodeFunction
func = CType(element, EnvDTE.CodeFunction)
If (func.Access <> vsCMAccess.vsCMAccessPrivate And func.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
DocumentElements(document, func.Children)
Case vsCMElement.vsCMElementProperty
Dim prop As EnvDTE.CodeProperty
prop = CType(element, EnvDTE.CodeProperty)
If (prop.Access <> vsCMAccess.vsCMAccessPrivate And prop.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
DocumentElements(document, prop.Children)
Case vsCMElement.vsCMElementEvent
Dim evt As EnvDTE80.CodeEvent
evt = CType(element, EnvDTE80.CodeEvent)
If (evt.Access <> vsCMAccess.vsCMAccessPrivate And evt.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
DocumentElements(document, evt.Children)
Case vsCMElement.vsCMElementClass
Dim cls As EnvDTE.CodeClass
cls = CType(element, EnvDTE.CodeClass)
If (cls.Access <> vsCMAccess.vsCMAccessPrivate And cls.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
DocumentElements(document, cls.Children)
Case vsCMElement.vsCMElementStruct
Dim strct As EnvDTE.CodeStruct
strct = CType(element, EnvDTE.CodeStruct)
If (strct.Access <> vsCMAccess.vsCMAccessPrivate And strct.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
DocumentElements(document, strct.Children)
Case vsCMElement.vsCMElementDelegate
Dim dlg As EnvDTE.CodeDelegate
dlg = CType(element, EnvDTE.CodeDelegate)
If (dlg.Access <> vsCMAccess.vsCMAccessPrivate And dlg.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
DocumentElements(document, dlg.Children)
Case vsCMElement.vsCMElementEnum
Dim enm As EnvDTE.CodeEnum
enm = CType(element, EnvDTE.CodeEnum)
If (enm.Access <> vsCMAccess.vsCMAccessPrivate And enm.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
DocumentElements(document, enm.Children)
Case (vsCMElement.vsCMElementVariable)
Dim var As EnvDTE.CodeVariable

var = CType(element, EnvDTE.CodeVariable)
If (var.Access <> vsCMAccess.vsCMAccessPrivate And var.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
Case vsCMElement.vsCMElementNamespace
Dim nmspc As EnvDTE.CodeNamespace
nmspc = CType(element, EnvDTE.CodeNamespace)
DocumentElements(document, nmspc.Children)
                Case vsCMElement.vsCMElementInterface
Dim inter As EnvDTE.CodeInterface
inter = CType(element, EnvDTE.CodeInterface)
If (inter.Access <> vsCMAccess.vsCMAccessPrivate And inter.Access <> vsCMAccess.vsCMAccessProject) Then
DocumentElement(document, element)
End If
DocumentElements(document, inter.Children)
End Select

Next
Catch ex As Exception

End Try

End Sub

End Module

Automating the Checking of Standards and Best Practices

During the past few months I have been taking care of some developer "ecosystem" tasks at work. You know, all those tedious first-time tasks that nobody looks forward to doing but everyone should see the value of having them done. Things that increase a team's sustainability. Things like setting up a CI build, dealing with tests that have been ostracized for over a year, converting projects to the latest IDE and framework, making sure project files are set to create debug symbols properly, re-checking the "run code analysis on build" setting, deleting the slanderous comments that blame me - I'm sure you get the picture.

One of the more interesting challenges in this endeavour has been trying to create what is essentially automatically enforced Standards and Best Practices (S&BPs) to be used by the entire group. The IDE and Team System can assist in that enforcement with Code Analysis rules, Team System check-in policies, and automatically executed tests. However, if you're a .Net developer you probably know the disdain that some of us have for enforcing the full multitude of available Static Analysis rules. It's a fairly common occurrence to find the rules to be unchecked of the "run code analysis on build" setting to be turned off, or simply that developers suppress all violations with a highly tuned reflex of the mouse-button finger. Additionally, I've never seen anyone spend the time to fix all the code analysis warnings, because they don't "stop the line" by breaking the build.

One FxCop file for all Visual Studio Projects

There is a way to have a single shared FxCop rule file that all projects reference, that project files can't override, and that role-base security can be applied. And it's easy to implement.

  • Create a StaticAnalysis.targets file with the content below
  • Put the file somewhere convenient in TFS source control
  • Change the source control permissions of that file to be limited
  • Edit each csproj or vbproj file in notepad and add the following line just before the </Import Project="$(MSbuildBinPath)\Microsoft.CSharp.targets" /> tag - placement is important for overriding the specific project's Static Analysis settings (c# version shown here)
<import project="[Relative TFS Work Space Path to Project]\StaticAnalysis.targets">

  • Build
  • Make sure your build server will properly get the StaticAnalysis.targets file to the location you referenced in your vbproj or csproj
  • Check-In
  • Notify your team that they need to get latest of the code and the location where you checked in the StaticAnalysis.targets file
What's in the file

The top line includes compiler warning numbers that we want to be treated as errors so that violations of them will "stop the line" and not just be ignored. Most of the numbers below are for XML comment issues. If you want to treat these XML comment problems as errors, consider this macro to help you clean up existing issues. You can get the number from warning statements in the output window when you compile.

The second section is the list of FxCop rules turned on as a waning (+), on as an error (+!) or off (-).

The File


<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<WarningsAsErrors>1572,1591,0105,1574,0649</WarningsAsErrors>
<CodeAnalysisRules>
+!JBBrown.FxCop.Rules#JB1502;
+!JBBrown.FxCop.Rules#JB1505;


+!Microsoft.Usage#CA2209;
+!Microsoft.Usage#CA2215;
+!Microsoft.Usage#CA2202;
+!Microsoft.Usage#CA2241;
+!Microsoft.Design#CA1012;
+!Microsoft.Design#CA2210;
+!Microsoft.Design#CA1040;
+!Microsoft.Design#CA1005;
-!Microsoft.Design#CA1020;
+!Microsoft.Design#CA1021;
+!Microsoft.Design#CA1010;
+!Microsoft.Design#CA1011;
+!Microsoft.Design#CA1009;
+!Microsoft.Design#CA1050;
+!Microsoft.Design#CA1026;
+!Microsoft.Design#CA1019;
+!Microsoft.Design#CA1031;
+!Microsoft.Design#CA1047;
+!Microsoft.Design#CA1000;
+!Microsoft.Design#CA1048;
+!Microsoft.Design#CA1051;
+!Microsoft.Design#CA1002;
+!Microsoft.Design#CA1061;
+!Microsoft.Design#CA1006;
+!Microsoft.Design#CA1046;
+!Microsoft.Design#CA1045;
+!Microsoft.Design#CA1038;
+!Microsoft.Design#CA1008;
+!Microsoft.Design#CA1028;
+!Microsoft.Design#CA1004;
+!Microsoft.Design#CA1035;
+!Microsoft.Design#CA1063;
+!Microsoft.Design#CA1032;
+!Microsoft.Design#CA1023;
+!Microsoft.Design#CA1033;
+!Microsoft.Design#CA1039;
-!Microsoft.Design#CA1016;
+!Microsoft.Design#CA1014;
+!Microsoft.Design#CA1017;
+!Microsoft.Design#CA1018;
+!Microsoft.Design#CA1027;
+!Microsoft.Design#CA1059;
+!Microsoft.Design#CA1060;
+!Microsoft.Design#CA1034;
+!Microsoft.Design#CA1013;
+!Microsoft.Design#CA1036;
+!Microsoft.Design#CA1044;
+!Microsoft.Design#CA1041;
+!Microsoft.Design#CA1025;
+!Microsoft.Design#CA1052;
+!Microsoft.Design#CA1053;
+!Microsoft.Design#CA1057;
+!Microsoft.Design#CA1058;
+!Microsoft.Design#CA1001;
+!Microsoft.Design#CA1049;
+!Microsoft.Design#CA1054;
+!Microsoft.Design#CA1056;
+!Microsoft.Design#CA1055;
+!Microsoft.Design#CA1030;
+!Microsoft.Design#CA1003;
+!Microsoft.Design#CA1007;
+!Microsoft.Design#CA1043;
+!Microsoft.Design#CA1024;
+!Microsoft.Design#CA1062;

-Microsoft.Globalization#CA1301;
-Microsoft.Globalization#CA1302;
-Microsoft.Globalization#CA1303;
-Microsoft.Globalization#CA1304;
-Microsoft.Globalization#CA1305;
-Microsoft.Globalization#CA1306;
+!Microsoft.Globalization#CA2101;
-Microsoft.Globalization#CA1300;

-!Microsoft.Interoperability#CA1403;
-!Microsoft.Interoperability#CA1406;
-!Microsoft.Interoperability#CA1413;
-!Microsoft.Interoperability#CA1402;
-!Microsoft.Interoperability#CA1407;
-!Microsoft.Interoperability#CA1404;
-!Microsoft.Interoperability#CA1410;
-!Microsoft.Interoperability#CA1411;
-!Microsoft.Interoperability#CA1405;
-!Microsoft.Interoperability#CA1409;
-!Microsoft.Interoperability#CA1415;
-!Microsoft.Interoperability#CA1408;
-!Microsoft.Interoperability#CA1414;
-!Microsoft.Interoperability#CA1412;
-!Microsoft.Interoperability#CA1400;
-!Microsoft.Interoperability#CA1401;

-!Microsoft.Mobility#CA1600;
-!Microsoft.Mobility#CA1601;

+!Microsoft.Naming#CA1700;
+!Microsoft.Naming#CA1712;
+!Microsoft.Naming#CA1713;
+!Microsoft.Naming#CA1709;
+!Microsoft.Naming#CA1708;
+!Microsoft.Naming#CA1715;
+!Microsoft.Naming#CA1710;
+!Microsoft.Naming#CA1720;
-!Microsoft.Naming#CA1707;
+!Microsoft.Naming#CA1722;
+!Microsoft.Naming#CA1711;
+!Microsoft.Naming#CA1716;
+!Microsoft.Naming#CA1725;
+!Microsoft.Naming#CA1719;
+!Microsoft.Naming#CA1721;
+!Microsoft.Naming#CA1724;
+!Microsoft.Naming#CA1726;

+!Microsoft.Maintainability#CA1501;
+!Microsoft.Maintainability#CA1500;
-Microsoft.Maintainability#CA1502;
-Microsoft.Maintainability#CA1505;

+!Microsoft.Performance#CA1809;
+!Microsoft.Performance#CA1811;
+!Microsoft.Performance#CA1812;
+!Microsoft.Performance#CA1813;
+!Microsoft.Performance#CA1823;
+!Microsoft.Performance#CA1800;
+!Microsoft.Performance#CA1805;
+!Microsoft.Performance#CA1810;
+!Microsoft.Performance#CA1822;
+!Microsoft.Performance#CA1815;
+!Microsoft.Performance#CA1814;
+!Microsoft.Performance#CA1819;
+!Microsoft.Performance#CA1804;
+!Microsoft.Performance#CA1820;
+!Microsoft.Performance#CA1802;
+!Microsoft.Performance#CA1807;
+!Microsoft.Performance#CA1817;
+!Microsoft.Performance#CA1818;

-!Microsoft.Portability#CA1901;
-!Microsoft.Portability#CA1900;

+!Microsoft.Reliability#CA2002;
+!Microsoft.Reliability#CA2003;
+!Microsoft.Reliability#CA2004;
+!Microsoft.Reliability#CA2006;
+!Microsoft.Reliability#CA2000;

+!Microsoft.Security#CA2116;
+!Microsoft.Security#CA2117;
+!Microsoft.Security#CA2105;
+!Microsoft.Security#CA2115;
+!Microsoft.Security#CA2104;
+!Microsoft.Security#CA2122;
+!Microsoft.Security#CA2114;
+!Microsoft.Security#CA2123;
+!Microsoft.Security#CA2111;
+!Microsoft.Security#CA2108;
+!Microsoft.Security#CA2107;
+!Microsoft.Security#CA2103;
+!Microsoft.Security#CA2118;
+!Microsoft.Security#CA2109;
+!Microsoft.Security#CA2119;
+!Microsoft.Security#CA2106;
+!Microsoft.Security#CA2112;
+!Microsoft.Security#CA2120;
+!Microsoft.Security#CA2121;
+!Microsoft.Security#CA2126;
+!Microsoft.Security#CA2124;
+!Microsoft.Security#CA2100;

+!Microsoft.Usage#CA2236;
+!Microsoft.Usage#CA1816;
+!Microsoft.Usage#CA2227;
+!Microsoft.Usage#CA2213;
+!Microsoft.Usage#CA2216;
+!Microsoft.Usage#CA2214;
+!Microsoft.Usage#CA2222;
+!Microsoft.Usage#CA1806;
+!Microsoft.Usage#CA2217;
+!Microsoft.Usage#CA2212;
+!Microsoft.Usage#CA2219;
+!Microsoft.Usage#CA2201;
+!Microsoft.Usage#CA2228;
+!Microsoft.Usage#CA2221;
+!Microsoft.Usage#CA2220;
+!Microsoft.Usage#CA2240;
+!Microsoft.Usage#CA2229;
+!Microsoft.Usage#CA2238;
+!Microsoft.Usage#CA2207;
+!Microsoft.Usage#CA2208;
+!Microsoft.Usage#CA2235;
+!Microsoft.Usage#CA2237;
+!Microsoft.Usage#CA2232;
+!Microsoft.Usage#CA2223;
+!Microsoft.Usage#CA2211;
+!Microsoft.Usage#CA2233;
+!Microsoft.Usage#CA2225;
+!Microsoft.Usage#CA2226;
+!Microsoft.Usage#CA2231;
+!Microsoft.Usage#CA2224;
+!Microsoft.Usage#CA2218;
+!Microsoft.Usage#CA2234;
+!Microsoft.Usage#CA2239;
+!Microsoft.Usage#CA2200;
+!Microsoft.Usage#CA1801;
+!Microsoft.Usage#CA2205;
+!Microsoft.Usage#CA2230;
</codeanalysisrules>
<RunCodeAnalysis>true</RunCodeAnalysis>
</PropertyGroup>
</Project>

Monday, September 1, 2008

Team System MVPs press on TeamReview

Over the last few weeks a couple of Team System MVPs, Omar Villarreal and Willy-Peter Schaub have been doing TeamReview a tremendous favor. Omar has been giving some great feedback on the TeamReview product and helped identify and resolve a few bugs. Additionally he was able to get Willy-Peter's attention which lead to a wonderful write-up of TeamReview and an outstanding WorkFlow guidance poster.

From Omar Villarreal on using TeamReview

From Willy-Peter Schaub

Thanks Omar and Willy-Peter!