paper_30_measure.tex

\documentclass[SDSUThesis.tex]{subfiles} 
\begin{document}

\section{MEASURING AN SDO}

    Anything can be measured \cite{Hubbard2010}.  Thus, an SDO can be 
    measured.  Proper measurement is crucial for improvement because
    without a starting point it is impossible to determine progress.
    Also, consistent reporting is essential for 
    tracking historical performance.  
    
    The SDLC, like any process, needs to be properly measured.  In
    order to accomplish proper measurement, three activities need
    to occur \cite{florac1999}.  
    
    \begin{enumerate}
        \item Identify Process Issues
        \item Select And Define Measures
        \item Integrate with the Software Process
    \end{enumerate}
    
    This dissertation will focus on steps 1 and 2.  The process issue is the overall
    effectiveness of the SDO.  Section \ref{sec:CRI} will cover step 2 as
    it relates to an SDO.  Step
    3 will be different for each SDO, but Section \ref{sec:SDLC-AE} provides
    a bit of guidance for storing the correct information.  It is up to the
    specific SDO to determine how and when the information is being stored.

    Many methods have been  used in the past 
    to measure and evaluate SDOs.  Some of the common methods
    will be explained in the following sections. 

    \subsection{METRICS}
        A \textit{metric} can be defined as a means of telling a complete story
        for the purpose of improving something \cite{Klubeck2011}.  Metrics are
        frequently indirect measurements and
        are very common in the measurement of SDOs.  
        The following are some examples of metrics that can be collected 
        for an SDO. 
        \begin{itemize}
            \item SLOC - The number of Source Lines of Code 
                \nomenclature{SLOC}{Source Lines Of Code}
            \item NOM - The Number of Methods per class
                \nomenclature{NOM}{Number of Methods}
            \item Complexity - A numerical measure of the code complexity
                (some common examples are McCabe \cite{McCabe1976} and 
                Halstead \cite{Halstead1977})
            \item Design - The amount of coupling and cohesion present 
                in the software code
            \item Source Code Analysis - Tools that determine whether 
                code adheres to specified set of rules. Common 
                examples are PMD\footnote{PMD is a source code analysis product.  
                It is not an acronym.} and FindBugs$^{TM}$ \cite{PMD, Findbugs}.
        \end{itemize}
        All these metrics are beneficial, but none of them tell the story of the entire
        SDO.  Most of the metrics for an SDO, as seen in the list above, focus on 
        the source code and development phase. 
        Since metrics are indirect, it can be very difficult to match SDO performance with
        a metric or series of metrics.  Metrics are great for tracking, but decision making
        based upon metrics alone is difficult.  That is why many of the other techniques
        build upon metrics to provide a more complete overall picture of an SDO. Metrics are a great starting point, but more is needed to properly 
        evaluate performance. 

    \subsection{INDICATORS}
    \label{sec:indicators}
        Another common measurement technique is indicators. An
        \textit{indicator} is simply a performance measure. 
        Typically, a number of indicators will be placed
        together and displayed in some report or on some
        dashboard. 
        Indicators can be crucial measurements
        within any business setting, and an SDO is no exception. Determining
        the correct indicators for an organization can
        be difficult, and many organizations incorrectly classify
        the indicators \cite{parmenter2010}.  
        The differences between the indicators will be explored and
        possible measures for each indicator in an SDO will be
        presented.   The four
        categories of indicators important to an SDO are shown
        in Table \ref{tab:indicators}.
        
        \nomenclature{RI}{Result Indicator}
        \nomenclature{KRI}{Key Result Indicator}
        \nomenclature{PI}{Performance Indicator}
        \nomenclature{KPI}{Key Performance Indicator}
        
        \begin{longtable}{@{}l l}
            \toprule%
             \centering%
             {\bfseries Performance Measure}
             & {\bfseries Description}\\
            
            \cmidrule[0.4pt](r{0.125em}){1-1}%
            \cmidrule[0.4pt](lr{0.125em}){2-2}%
            % \midrule
            \endhead
            
            RI (Result Indicator) & What has been done?   \\
            \myrowcolour%
            KRI (Key Result Indicator) & How you have done? \\
            PI (Performance Indicator) & What to do? \\
            \myrowcolour%
            KPI (Key Performance Indicator) & How to dramatically increase performance? \\
            
            \bottomrule
            
            \caption{INDICATORS}
            \label{tab:indicators}
        \end{longtable}
            
        Indicators can be used in just about every organizational
        setting from businesses to non-profit organizations. They
        are not unique to SDOs, and the exact indicators to track
        are very specific to the organization.  The indicators chosen
        by one organization might not be the same as the indicators
        chosen by another organization.  The following sections
        will explain the type of indicators in more detail and provide
        some examples for an SDO.
        
        \subsubsection{RESULT INDICATORS (RI) FOR AN SDO}
            \textit{Result Indicators} are performance 
            measures that summarize activity.  All financial performance
            measures are result indicators.  Result indicators
            are measured on a timely basis (daily, weekly, monthly)
            and are the result of more than one activity.  They do not
            tell staff what needs to be done to improve the RI. For
            an SDO, some possible RIs are seen in Table \ref{tab:RI}.
            
            \begin{longtable}{@{}l l}
                \toprule%
                 \centering%
                 {\bfseries Result Indicators} &
                 \\
                
                \cmidrule[0.4pt](r{0.125em}){1-1}%
                % \midrule
                \endhead
                
                Requirements Implemented Per Month  \\
                \myrowcolour%
                Monthly SLOC \\
                New Weekly Users \\
                \myrowcolour%
                Monthly Development Hours \\
                Webpage Views Yesterday \\
                \myrowcolour%
                Monthly Development Hours \\
                Monthly Server Uptime \\
                \myrowcolour%
                Quartely Software Sales \\
                
                \bottomrule
                
                \caption{RESULT INDICATORS FOR AN SDO}
                \label{tab:RI}
            \end{longtable}
            
        \subsubsection{KEY RESULT INDICATOR (KRI) FOR AN SDO}
            \textit{Key Result Indicators} are measures of multiple
            activities that give a clear picture of whether the
            organization is traveling in the right direction.  Unfortunately,
            KRIs are commonly mistaken for KPIs \cite{parmenter2010}.  KRIs
            do not tell an organization what is needed to improve the 
            results.  KRIs are highly beneficial for high-level management
            and not necessarily beneficial for staff working directly on 
            the software.  For
            an SDO, some possible KRIs are seen in Table \ref{tab:KRI}.
            
            \begin{longtable}{@{}l l}
                \toprule%
                 \centering%
                 {\bfseries Key Result Indicators} &
                 \\
                
                \cmidrule[0.4pt](r{0.125em}){1-1}%
                % \midrule
                \endhead
                
                Customer Satisfaction  \\
                \myrowcolour%
                Net Profit \\
                Money Spent on Fixing Software \\
                \myrowcolour%
                \% of New Features vs. Fixes \\
                Time on Website \\
                \myrowcolour%
                \% Servers Meeting the Expected Availability \\
                
                \bottomrule
                
                \caption{KEY RESULT INDICATORS FOR AN SDO}
                \label{tab:KRI}
            \end{longtable}
            
        
        \subsubsection{PERFORMANCE INDICATOR (PI) FOR AN SDO}
            \textit{Performance Indicators} are non-financial performance measures
            that help a team align themselves with the organizations 
            strategy.  These are important to the organizations success, but
            they are not the key measures that will lead to drastic improvement.  They are specifically tied to a team and all staff understand
            what actions need to be taken to improve the PI.
            For an SDO, some possible PIs are seen in Table \ref{tab:PI}.
                
            \begin{longtable}{@{}l l}
                \toprule%
                 \centering%
                 {\bfseries Performance Indicators} &
                 \\
                
                \cmidrule[0.4pt](r{0.125em}){1-1}%
                % \midrule
                \endhead
                
                \% Test Coverage\footnote{Test Coverage is simply the percentage 
                    of the code that is being tested. Ideally, this 
                    number would be 100\%, but higher is better.}   \\
                \myrowcolour%
                \# of Project Defects from Key Customers  \\
                Requirements Scheduled for Next Release  \\
                \myrowcolour%
                \# of Missed Requirements \\
                \% of Late Projects \\
                
                \bottomrule
                
                \caption{PERFORMANCE INDICATORS FOR AN SDO}
                \label{tab:PI}
            \end{longtable}
            
        \subsubsection{KEY PERFORMANCE INDICATOR (KPI) FOR AN SDO}
            \textit{Key Performance Indicators} are performance measures
            focusing on critical aspects for current and future organizational
            success.  Notice, KPIs are not focused on historical performance,
            and they clearly indicate how to drastically increase performance.
            KPIs allow a team to monitor current performance and 
            quickly take action to correct future performance.
            KPIs cover a shorter time frame than KRIs.
            KPIs consist of the following seven characteristics \cite{parmenter2010}.
            \begin{enumerate}
                \item Not financial
                \item Measured frequently (hourly, daily, weekly)
                \item Acted on by CEO\footnote{Chief Executive Officer} 
                    and/or upper-level management 
                    \nomenclature{CEO}{Chief Executive Officer}
                \item Clearly indicate the action required
                \item Tie responsibility to a particular team
                \item Have a significant impact
                \item Encourage appropriate action
            \end{enumerate}
            
            Antolic in \cite{Antolic2008} made one of the earliest attempts
            to identify and measure the KPIs for an SDO. Antolic's strategy
            focused around six KPIs.
            \begin{enumerate}
                \item Schedule adherence
                \item Assigned content adherence
                \item Cost adherence
                \item Fault slip through
                \item Trouble report closure rate
                \item Cost per defect
            \end{enumerate}
            However, according to the definition of KPI just presented,
            they are not really KPIs but instead KRIs.  That is because
            none of the six clearly indicate the action required to 
            improve the measure.  Also, it is unclear if improving any of
            the measures will drastically improve performance.
            
            For an SDO, Table \ref{tab:KPI} identifies some more appropriate
            KPIs.  
            
            \begin{longtable}{@{}l l}
                \toprule%
                 \centering%
                 {\bfseries Key Performance Indicators} &
                 \\
                
                \cmidrule[0.4pt](r{0.125em}){1-1}%
                % \midrule
                \endhead
                
                Projects more than 20\% behind schedule \\
                \myrowcolour%
                Servers currently unavailable \\
                Automated tests failing for more than 24 hours \\
                \myrowcolour%
                Projects with test coverage less than 60\% \\
                Projects with more than 10 SIT defects \\
                \myrowcolour%
                Unfixed, high priority PROD defects older than 1 week  \\
                
                \bottomrule
                
                \caption{KEY PERFORMANCE INDICATORS FOR AN SDO}
                \label{tab:KPI}
            \end{longtable}
    
    \subsection{BALANCED SCORECARD}
    \label{sec:bsc}
    
        Developed in 1992 by Robert S. Kaplan and David P. Norton, the 
        \textit{balanced scorecard}  is a set of measures that
        give management a quick and comprehensive view of the
        organization \cite{kaplan1992}.  Originally created as an
        extension to the already existing financial measures, the
        balanced scorecard expanded the measures to include: 
        customer focus, internal process, and learning/growth. 
        This gave organizations a more comprehensive view that was
        not strictly financial.  It allows an organization to focus
        on long-term strategic goals instead of just short-term goals.
        As a result of the strategic focus, the balance scorecard
        rapidly gained widespread adoption
        among businesses \cite{Kaplan2007}.  
        
        In 2010, David Parmenter \cite{parmenter2010} added two more
        characteristics to the balanced scorecard: employee satisfaction
        and environment/community.  This results in a total of six 
        characteristics for the balanced scorecard.
        \begin{enumerate}
          \item Financial 
          \item Customer Focus
          \item Internal Process
          \item Learning and Growth
          \item Employee Satisfaction\footnote{\label{ft:bsc} Added 
          later by Parmenter \cite{parmenter2010}. }
          \item Environment/Community\textsuperscript{\ref{ft:bsc}}
        \end{enumerate}
        
        Balanced scorecards are great for easily displaying the 
        important information about
        an organization.  The downside is a balanced scorecard does not specify what
        exactly needs to be tracked. It can be very difficult to determine exactly
        what PIs, RIs, KRIs and/or KPIs to track in a balanced scorecard. 
        It specifies six broad categories.
        It is also not specific to an SDO and 
        it does not produce a single number.  However, any new measurement
        technique for an organization should be compared with the balanced
        scorecard.  

    \subsection{PROJECT MANAGEMENT MEASUREMENT}
    \label{sec:pm}
        A final strategy to measure an SDO is focused on the aspect of
        project management. Project management is the guidance
        applied to a project to ensure an effective and efficient
        completion.  Proper project management will ensure all steps
        of the SDLC
        continue to progress and all obstacles are handled
        in a timely fashion.  
        According to Putnam and Myers in \cite{Putnam2013}, 
        the five core measurements for managing software
        projects are:
        
        \begin{enumerate}
            \item \textbf{Quantity of function} - usually measured in terms of size (such as source lines of code), that ultimately execute on the computer
            \item \textbf{Productivity} - as expressed in terms of the functionality produced for the time and effort expended
            \item \textbf{Time} - the duration of the project in calendar months
            \item \textbf{Effort} - the amount of work expended in person-months
            \item \textbf{Reliability} - as expressed in terms of defect rate (or its reciprocal, mean time to defect)
        \end{enumerate}
        
        A \textit{process productivity} number is calculated based entirely
        on aspects of the SDLC and the five core measurements.  It is a number
        targeted at project teams working on the SDLC.  
       
    \subsection{A SIMPLER MEASUREMENT}
    
        It is important to note that SDOs do not just
        develop software.  An SDO has many other duties
        including: deploying software, installing server hardware/software,
        writing documentation, surveying users, research, innovation,
        education and other common business duties.  Thus it is
        important to measure as many duties as possible.  
    
        How can the PIs, RIs, KRIs, and KPIs be combined to form a single
        value called the CRI (Cumulative Result Indicator)?  If the indicators
        are targeted for upper management to understand performance, then 
        KPIs are not the correct indicators.  KPIs are targeted towards
        immediate action and future performance.  RIs and KRIs are the most
        beneficial for upper management to gauge how an organization is doing.
        However, with so many possible RIs and even KRIs, it can be tricky
        to gain a quick understanding.  The next section will present
        and explain a technique to combine KRIs into a single number
        for immediate and  effortless evaluation of an SDO.
        
\end{document}