Towards Data-Driven Competencies – Part 2
Part 1 of this short blog series discussed the nature of #competencies and asked the question: what combination of knowledge, skills, abilities and other attributes does a person need to be competent at doing a certain thing, at a certain level?
In this post I’ll talk about a technique we’re developing at Blue Mirror Insights that matches competencies from different sources by looking for common underlying elements of skill, knowledge and ability. This is particularly important in some of the work we’re doing at the moment to measure the learning outcomes of degree course curricula and help answer the question asked by students: how employable will this course make me?
Example: Seeing the Big Picture
In our work at BMI, we have surveyed many employer-side competency frameworks – some of these are mentioned in Part 1 of this blog series. Here are two examples of competencies as described by two highly-respected organisations:
UK Civil Service: Seeing the big picture
Seeing the big picture is about having an in-depth understanding and knowledge of how your role fits with and supports organisational objectives and the wider public needs and the national interest. For all staff, it is about focusing your contribution on the activities which will meet Civil Service goals and deliver the greatest value. For leaders, it is about scanning the political context and taking account of wider impacts to develop long term implementation strategies that maximises opportunities to add value to the citizen and support economic, sustainable growth.
Organisation for Economic Co-operation and Development (OECD): Organisational knowledge
The ability to understand the power relationships within an organisation and with other organisations. It includes the ability to understand the formal rules and structures including the ability to identify who the real decision-makers are as well as the individuals who can influence them.
Intuitively, these seem like competencies that will demand similar combinations of attributes from employees. But how can we use data to support this observation?
One answer – and the one that underpins our development work at BMI – is to match the descriptions of each competency and match it against a library of common attributes. You can think of this as ‘googling the competencies’.
To do this, we use the world-renowned Occupational Information Network (O*NET #onet) database from the US (https://www.onetonline.org/) and combine this with a specialised Natural Language Processing (NLP) software engine. The text engine looks for the meaning in each sentence of the competency descriptions and returns a relevancy score against each matching O*NET attribute.
In this data visualisation, we show the relevancy paths for the two competencies we introduced above, when matched to the O*NET Work Activity attributes. This is equivalent to asking the question: what might a person with this competency be doing in their job?
Whilst some outliers can be identified in the results, we can see how, depending on their seniority, people would need a competent understanding of their own organisation to be involved in:
Consulting with, advising, training and/or assisting fellow employees;
Developing corporate or sub-corporate objectives and strategies;
Getting, processing and analysing organisational information;
Resolving organisational conflicts;
Organisational-level decision making and/or problem solving.
Taking this a step further, we can run a basic cluster analysis over all the UK Civil Service and OECD competencies to see which competencies might lead to people doing similar things in the course of their job. Note that our two organisational knowledge competencies sit in cluster 7 – interestingly enough with the OECD Analytical thinking competency, which might indicate that seeing the big picture in organisations like OECD and the UK Civil Service does take a bit of thinking about!
Matching Supply to Demand: the Educational Dimension
We can, of course, apply the same process to the equivalent competencies as taught in universities. However, these rarely exist in the same form as in the workplace – after all, how can a teaching institution hope to gear its education to employer competency requirements when there is no single consensus?
Increasingly though, universities are describing graduate attributes as skills and mindsets that are acquired by students through the teaching and activities offered.
Two well-developed sets of graduate attributes that we have reviewed come from the universities of Edinburgh and Glasgow. With a little data wrangling combined with professional judgement and validation, the descriptions of these graduate attributes may arguably be used in the same way as we described previously.
Once this is done, we can then match acquired graduate attributes (‘supply’) to required employer competencies (‘demand’) using the common underlying O*NET attributes from each side.
The results should be interpreted carefully, but the principle here is that we are looking for graduate attributes that share at least one underlying O*NET attribute with each employer competency. This by no means implies that a graduate scoring highly in a particular graduate attribute can be thought of as having the matched employer competency. However, it may help.
As an example, in the data visualisation shown above, we can see that confidence in a graduate from the University of Glasgow is likely to help with pretty much everything required from both the OECD and the UK Civil Service. Intuitively, we may agree with that. Reflective learning is possibly less important to employers, although an inspection of the University’s description of this attribute shows that this includes an understanding of the importance of continuing personal/professional development, which is of course a valued mindset for many organisations.
Finding the right data
A note about the data we’re using. The technique described in this blog post uses the O*NET Generalised Work Activity library as the common underlying elements against which employer competencies and graduate attributes are matched. We acknowledge that this is open to debate. As I said in Part 1, BMI accepts the concept of a competency as a ‘cluster’ of knowledge, skills and other attributes that lead to successful performance. In this example, the O*NET Generalised Work Activities are used here as a proxy for this definition of competencies in order to illustrate the methodology.
In the following posts in this series, I’ll explore further alternative sources of data that we may use to underpin our methodology.
Footnote: Why are we doing this?
In a recent post from my colleague Dr Michael Cross, Michael comments on a proposal by the UK’s Universities Minister for a ratings scheme for university courses. Michael’s comment is that instead of a scheme derived in some (arguably obscure) way from data such as student satisfaction and drop out rates, perhaps it might be better to look at the #employability potential of each course.
In this blog post, I’ve described some of BMI’s early development work in finding a methodology to level the playing field between employers and education by using a common language and an employability matching process between the two sides. We’re exposing our thinking at this early stage because we feel that there is an appetite for this in the current political and economic climate, including the ratings scheme mentioned above.
We therefore invite comment and collaboration on our work, including offers for core funding from interested parties.