A lot of thought went into the method used behind the Risk Assessment Tool for IT Projects which I built. Unlike the many others on the market which are just marketing front doors, I wanted mine to be accurate and useful for the project managers and risk managers out there. And to do that, some serious design thought was called for.
In this article, I want to explain a bit more about the method behind the tool and how it calculates the risk ratings. I also want to throw open the method I chose for critique and discussion in an attempt to improve.
Recap: Risk Ratings Explained
Feedback from our early beta testing of the risk assessment tool indicated that users would like to understand what the range of possible ratings were, along with a short description of each one. So we incorporated this into the tool, of which you can see here: Risk Ratings Explained
Over time, I started to get more requests for how the risk ratings were calculated. This is because our selection of early beta testers were largely digital project managers who were interested in what to do to avoid project risks, rather than individuals with a strong risk background who also need to understand if the basis upon which the tool works is sound.
So, here’s how the tool works under the covers…
Assumptions & Terminology
Firstly, the risk assessment tool was built with the following in mind:
- A specific applicability to IT projects between £1M- £10M in size
- There are factors, which when present in a project, elevate the risk of a project going wrong
- Not all factors have the same effect with some increasing the threat to a project more than others
- Regarding the risk assessment tool, we refer to these factors as “threats” given the broad meaning of the term
- Threats are not project risks or issues (as you typically know it)
- Each threat, if applicable, uncovers a broad area of concern and will require further analysis to better understand what is actually going on. One or more actual project risks (or issues) may then come out of this analysis.
Question: Why not just ask risk related questions rather than introduce the idea of “threats” ? (which are a bit fuzzy and cannot simply be put onto a RAID log)
Answer: I certainly could have done and indeed many other tools take this approach (example). KPMG has such a tool and in it, there are circa 2000 questions. In this modern day and age of fast, fast, fast, I have observed from my site logs the difficulty of getting visitors to simply complete the first 15 mandatory questions. Can you imagine how many would actually complete 2000 questions…
Step 1. Hierarchy of threats
Threats are “things” in and around the project environment which influence the workings of the project (or temporary organisation). There are many different types of threats and we visualise them as being able to live in one of several different “tiers”, namely:
It is our belief that a risk assessment tool must holistically address threats in each tier if it is to stand a chance at being accurate and useful in identifying potential project issues.
Step 2. Identifying threats in each tier
The tool has 45 questions which results in a total of 145 recommendations covering what to consider doing in order to address each applicable threat.
The questions have been carefully chosen to broadly cover all the “tiers” in the previous diagram, and have been grouped into 17 different categories (to help with user friendly presentation aspects in the tool). The categories are as follows:
|Category||Number of Questions|
|Procurement & Supplier Management||5|
|Industry & Business Landscape||3|
|Deployment and Post “go-live”||2|
|External Dependencies / 3rd Parties||1|
|Project Ways of Working||1|
Step 3. Relative score of each threat
We already knew that each threat doesn’t necessarily pose the same heightened risk as another one. So we needed to account for this in the risk assessment model somehow.
Luckily there is reliable evidence that shows human beings can make good relative estimates when asked, compared to making absolute estimates.
For example, “is today hotter or colder than yesterday?” is more likely to get an accurate answer compared to framing the question as “what is the temperature today in celsius?”
We took each question out of the 45 in total and scored it’s expected level of heightened risk / effect on project management / increased complexity in a relative fashion against all other 44 threats.
We also used the exponentially increasing Fibonacci series, 1 2 3 5 8 13 21 34 55, as the basis of scoring, so that we could assign a relative score that was an increase / decrease in an exponential way compard to its neighbours either side.
This allowed us to 1). exploit the fact that human beings can make good relative estimates, and 2). reduce the margin of error in the estimate.
For example, and to illustrate how this worked in practice, here are 3 threat entries from the risk assessment model which have been relatively scored, along with the rationale as to the score assigned:
|Reputational damage if the system does not work correctly||55||Impact of occuring can be very damaging and hard to recover from – total project effort to avoid this scenario is often extreme|
|Business critical operation with very high level of uptime||8||Can be achieved, but requires excellent design, careful consideration and adequate funding|
|Key person dependency exists||3||Mitigations are relatively easy if well planned|
It’s important to remember that the relatively scoring exercise was performed with the following assumption in mind, A specific applicability to IT projects between £1M- £10M in size, based on our personal experience of managing a wide variety of these types of IT projects.
We are fairly comfortable with the relative scores assigned as a starting point for the risk assessment model, because we further analysed the final scoring and found it to loosely follow a normal distribution, which is what we would expect:
Step 4. Total score of a given risk assessment
Following on from our earlier example, lets assume the person using the risk assessment tool answered Yes to the following three questions:
|Would the company / organisation suffer reputational damage should the system not work correctly?||55|
|Is the system business critical and will require a very high level of uptime?||8|
|Would the sudden departure of any team member cause significant issues?||3|
The threat score calculated by the risk assessment model would be 66 (ie. 55 + 8 + 3).
Given the relative scores across all 45 questions total 985, then this assessment would be calculated as having 6.7% of the total threat avalable to be allocated.
Step 5. Converting the total score to a risk rating
We now need to convert the Threat % into a user friendly risk rating, as per defined on the following page: Risk Ratings Explained. We do this because our testing showed that most project managers want a simple way to understand how much risk they are carrying.
Unfortunately this is also where it gets a bit tricky and prone to error. Risk and threat modelling is a non-linear business. It it also notoriously difficult to get right.
We came up with the following cutoffs based on our own personal experience of managing a wide variety of IT projects and also analysing a dataset with many different risk assessments to fine tune how a percentage threat score is finally turned into a risk rating.
|Risk Rating||Threat % |
Carrying on with our example, 6.7% of Threat would result in a risk rating of MEDIUM.
We are well aware of the limitations of the above approach, however we feel it’s reasonable (at this point in time) given two things:
- Many project managers in charge of IT projects are going about risk management in ways far inferior to what our tool can provide them, even with the limitations of the above approach [nb. granted this will not be true for experienced, enterprise risk managers with excellent tooling, for example]
- The risk assessment tool has an ongoing roadmap to further develop and improvement the accuracy of the tool in every sense (see the following sections for more detail).
We have a few ideas that we are planning to develop during 2020 and beyond to increase the usefuless and accuracy of the risk assessment tool, for example:
- Refine the relative scoring of questions via an expert panel (large enough to smooth out any local differences of opinion)
- Introducing partial answers to each question (ie. Yes, No, Often, Sometimes, NA)
- Some mathematical “curve fitting” to ensure the calculated risk ratings are properly distributed
- Correlation between the occurrence of multiple threats
- Being able to update a given risk assessment over time – and see how the risk rating has changed as the project has progressed through different project stages
- User accounts so that you have a full history of all assessments performed
Question: Are you using a validated approach?
Short answer: No
Longer answer: Not yet
The end game for the risk assessment tool is a validated model and underlying dataset, so that threat identification and recommendations are benchmarked and can be proven to work within a certain accuracy / confidence interval.
Whilst a multi-year project and one with some significant hurdles to overcome, we nontheless believe this is a possible and worthy vision to head towards. All future tool development is made with this ultimate end in mind.
The future and beyond
Wouldn’t it be great if there was a risk management tool that was so valuable, you decided to carry it around in your pocket from project to project. Knowing that you could reach for your IPhone / Android at any time and get accurate and validated project management help whenever you needed.
That is the future we are on the way to building.
Frank Ray & Associates is a software engineering consultancy that builds high quality software for businesses.