Everything that is wrong with ‘e-assessment’ in 2023. Corruption, poor quality technology, ineptitude, monopolies, we have it all. Because it’s not about the learners in e-assessment.
by VICTVS
Have you ever felt like you’re stuck in a bygone era? Have you ever wondered why, despite humanity’s incredible technological progress over recent decades, some things just never seem to change? Have you ever been in a position at work where your most critical task is dependent on an outdated, unscalable, inflexible piece of software that isn’t even supported by the manufacturer anymore? People who have to work with ‘e-Assessment’ technology certainly have…
As an industry, so called ‘eAssessment’ (electronic assessment for anyone who missed the 90s fad of adding an ‘e’ to things to make it sound snazzy and modern #eyeroll), software evolves at a rate comparable to the movement of the tectonic plates.
New features that would have seemed impressive in the mid-90’s, are lauded as though they are the keys to inter-stellar travel when in 2022, they would not impress a 10-year-old. It is a bizarre, phantom realm of science fiction where antiquated technologies of yesteryear collide with modern users and their modern expectations – with all of the entirely predictable confusion, fear and panic of any good episode of the Twilight Zone. If ever there was an industry subject to an inexplicable blackhole of innovation and creativity, ‘eAssessment’ is where that is! But I digress. Allow me to illustrate my point a little with the paragraphs below.
Please forgive me for failing to name names in this short article, but gentlemanly conduct forbids it. Whilst I do possess a low opinion of so many people in the industry for what I perceive to be their determined efforts to continually deceive and mislead customers, I would rather that they rise to the challenge of competition and genuine innovation, than simply head out on a mission to discredit them. Afterall, we are surely all in this to create the best possible experience for awarding organisations and their learners…
So, onto the problems at hand. What is wrong with ‘eAssessment’? This…
THE NAME
THE TECHNOLOGY
THE DEVELOPERS
THE CONSULTANTS
THE DESIGN
THE INDUSTRY BODIES
THE GOVERNMENT
CONCLUSION
THE NAME
Starting with the basics. ‘eAssessment’? Really? Electronic Assessment? That’s the type of thing that children have built into their strange electronic toys that people forget to give them batteries with at Christmas. The whole ‘e’ label is incredibly 90s. It’s as dated as the ‘I’ label will surely become, and it does not stand the test of time. Are modern, digital, cloud-based assessments being accurately described when they have the ‘eAssessment’ moniker applied to them? Possibly in a tangential sort of fashion whereby one could suggest that they are completed on devices that run on electricity, but really? Are the assessments electronic? So odd. Time for a change. This label has its slippers on. Digital Assessment is more accurate, more credible, and more genuine. Make the change!
THE TECHNOLOGY
In the 90s, anything was possible. So long as it wasn’t too advanced. Web services were not easily scalable because the infrastructure could not support global-scale assessment data transfer or massive numbers of concurrent users, and there was a reluctance to move over to digital from the hallowed ground of pen and paper. Websites looked terrible, hardware looked terrible and was terrible, and user experience was not a consideration as developers developed sites and resources for other developers – designers hadn’t got a hold on the new digital medium yet. Shell suits were everywhere, Edmonds was at the height of his popularity, and Johnny 5 offered an intriguing glimpse into the potential future of AI.
It was a golden era of technological optimism (when viewed through rose-tinted spectacles), but none of it worked very well. Back then, a 10-hour service outage would have been seen as an inevitable consequence of using such new-fangled nonsense, and global service blackouts were the norm. Answers as to what went wrong, were not demanded by clients because they understood so little in comparison to the ‘experts’ that they had hired to build their ‘e’ systems.
Which begs the question of why these things remain ubiquitous in 2022? Why the service failures? Why the terrible design? Why the reliance on old-fashioned methodologies? Some people in our industry would claim that it’s because this is what the clients want. They say that the awarding bodies are not interested in using new technologies and that they just want something that works. That may be the case but my point is that the outdated rubbish peddled by some, does not work and offers a user experience that is insulting to paying users. Whilst the buyers may want something rubbish, perhaps it is time that the candidates themselves had more of a say in the quality of the experience that they receive?
THE DEVELOPERS
Claiming that you’re a proudly British company with strong regional ties, whilst outsourcing your software development work (your main service offering) to Russia, is not cool. Especially when it’s all about cutting costs.
Stating that your software has delivered millions of tests, without explaining that lots of them have been technically disastrous and have failed to provide the expected experience, is not cool either.
Far too many tech development companies within the e-Assessment industry are comfortable with putting their client logos all over their webpages, trading on the success of the people they serve, even if their own products and services are woefully short of the mark. For example, claiming to be an innovative software development company whilst creating solutions that rely on local server downloads of assessment materials (actually requiring a physical device from which each user’s PC can obtain the assessment information in real time. Move over cloud computing…)
We also hear all the time, that developers will quote tens of thousands of pounds in fees, and require several weeks of development time, just to action a very minor change in their client systems. This is ridiculous and is exploitative of the fact that awarding bodies typically have limited in-house technical capability and understanding and cannot access and manipulate the code for themselves.
Outsourcing has also become common place within e-Assessment. Suppliers sell ‘their’ solutions to awarding bodies, only for it to turn out that they are selling someone else’s product as their own and that the party responsible for the development of the product is in an entirely different country. So imagine a large US firm selling you their assessment platform, only for it to turn out that it was developed by an Indian company, is maintained in India, and that what you have bought is not a bespoke product that represents the cutting edge of the industry, but is something that you, and the supplier, have very or no control over.
THE CONSULTANTS
Hourly rates for wishy-washy advice that is unspecific, unmeasurable, unachievable, irrelevant and untimely.
The business model of most consultants is based on time spent on a project, therefore it is in their interests to spend as much time as possible faffing around and not really helping anyone to reach a conclusion.
But when you have consultants, who also happen to be shareholders in key industry suppliers, and they are both shareholders and stakeholders in ‘industry bodies’ that claim to represent both buyers and suppliers, then you have a recipe for disaster. It’s like asking a doctor for medical advice when the doctor is paid commission by a pharmaceutical company for every prescription of opioids that they write (good idea US…).
So ok, consultants who have interests in the suppliers that they are supposedly independently of, is nothing new and it isn’t a distinctly 90s phenomenon, but it is rubbish, and it is letting down the industry. Personal allegiances and agendas should not affect the advice offered by any reputable consultant in any field, but unfortunately people are just people. Unlike the ruthless killing machine robots of the highly successful Terminator franchise, humans are subject to bias – even when we try not to be – and asking for advice from outside experts is a good idea as it helps to mitigate the risks caused by unintentionally biased thinking, but this can create other risks that people should be aware of.
To dwell on the point for a moment, there are consulting firms that are hired by awarding bodies to help them navigate their way through tendering exercises as they look for the best product or service to meet their needs. For example, this might be assessment software, remote proctoring services, test centre provision etc. The consultants market themselves as industry experts with decades of experience in the field, and as the best possible people for an awarding body to hire if they need informed, unbiased advice that will be justifiable to the awarding body directors, and that will vastly increase the chances of the solution meeting the requirements. What you then see in reality, is that the people who make up the team of ‘experts’ from the consultancy firms, may indeed have years of industry experience, but that this was gained whilst in the employ of one, or more, of the companies that are suppliers to the awarding bodies. Whilst this does not prevent a consultant from offering independent procurement advice, we know of consultants who have significant shareholdings in these companies. So they are part owners of, and are financially compensated by, companies whose products and services they are supposed to be offering unbiased advice on to their clients. We know who you are…
THE ‘DESIGN’
Enjoying all the gifs? We love them. But we wouldn’t put them into an ‘e-assessment’ or into a digital assessment either.
So many of the assessment platforms and other technologies that we come into contact within our work, look like antiques from the 90s that have been preserved as though they are relics to be displayed in the museum from Demolition Man. It’s 2022 and we keep having to assail our eyes with assessment software that looks as though it was designed for use on an Apricot PC.
Crafting effective design and user experience in a piece of software is an art form. It requires specialist attention – not some half-baked attempt by someone who fancies themselves as a ‘creative’.
Exams and assessments are hard enough already. Why make the experience worse for the candidate with terrible, counter-intuitive user controls, buttons that don’t make sense, fonts that require effort to read and essential features that are hidden where no right-minded individual would ever look…. This isn’t just getting the basics wrong, it’s a fundamental failure on the part of the suppliers because of the time lost by candidates and others, messing around trying to figure out an interface that makes no sense, and because of the stress that this causes to the candidates. Effective assessment of a candidate’s knowledge, does not occur when their opportunity to demonstrate what they have learnt, is subject to such unpredictable interference. For example, we know of one piece of software in which should a candidate put a comma rather than a decimal point between numbers (common practice outside the UK), the whole system crashes!
It’s called User Experience people. Get with it.
THE INDUSTRY BODIES
Conflict of interest anyone? Not so long as you declare it eh? Or why bother? Just everyone get into each other’s pockets, pretend that you give a monkey’s about what might be good for the interests of awarding bodies and students, whilst doing all that you can to ensure that your pals keep winning the contracts despite their inability to deliver anything on time or at the standard agreed. This is really tragic. Industry bodies have an opportunity to drive the continual innovation of best practice in their respective industry. They have the voice with which to represent small stakeholders and to prevent them from being drowned out by large corporates who can use their capital reserves to shout loudest through their marketing (hardly a new phenomenon this so it’s a shock that it seems like such an afterthought to these champions of our industry).
So instead of fair representation, innovation is overlooked because the attention falls on ‘platinum sponsors’ of events and so on. So key clients are only ever exposed to the big corporate names whilst all of the smaller suppliers face an uphill battle. But don’t get me wrong, I’m not complaining about have to fight for a seat at the table. The fact is that the largest players in the game, do lose. Regularly. Because they often fail to deliver to their clients, leaving a trail of devastation and disappointment behind them that can be impossible to recover from. So, they can get stuffed. But my point here is that any organisation claiming, or implying, that they are acting to promote best practice in our industry, and that they represent the widest possible group of stakeholders; probably shouldn’t be a privately owned company that has shareholders who are directly connected to well-known suppliers from within the same professional space. I mean, this isn’t oil and gas…
But what would you expect. Our industry is a refuge for oddballs including a failed politician who used to work with Tony Blair. This person recently claimed that Euan Blaire has done more than anyone else to revolutionise further education in the last 20 years. Gammon! His ridiculously hyped apprenticeship company (fake unicorn and obvious con) Multiverse is an almost unbelievable scam, and we know plenty of training providers who have done 1000 times more than Multiverse will ever do (before it’s sold to pensioners and widows as the next big thing before it’s inevitable implosion). Maybe they’ll book sales before they happen (like one particular e-assessment software provider), in a crafty throwback to Enron!
I know we live in an age where ‘cronyism’ is a word that has lost all weight with us, but let’s call it what it is – it’s corruption. And we have an industry body in our industry of education, that is led by someone who thinks it’s ok to be so openly corrupt, and such a grotesque toady, that they will publicly disrespect all of the legitimately awesome efforts of training providers throughout the UK, just so they can blow smoke up the bum of their former employer’s son. Genius.
THE GOVERNMENT
Hard not to name names here, but there is a government body that is supported by £1Bn of tax-payers money every year, that also makes hundreds of millions through exam delivery services, and yet routinely embarrasses the UK through its sheer incompetence.
For example, a few years ago, they paid £80,000.00 of public money (that’s our taxes by the way), to have their logo redesigned. That’s despite the union flag being free to use…
They currently owe the British taxpayers £270 million for loans during the lockdowns, and there are no plans for how this money will be repaid and they will not share details of the loan – despite this being our money.
The reason for them needing all this money? Their out-of-date business model of investing huge sums in real estate, terrible software platforms and technology that is redundant before it is even rolled out.
In a damning reflection on their incompetence, the UK’s leading financial services awarding body, that also acts as an advisor to the government, declared this particular organisation to be insolvent.
But still, they roam the Earth publicising pictures of their CEO interacting with people overseas, in what really comes across as another tragic hangover of the UK’s imperial past and that have a real vibe of the noble white man blessing the less fortunate with his presence.
Why does any of this matter
The reason that any of these complaints matter, is that this affects us all.
We are living through the dawn of the information age, which began in the 90s and continues today. This is fantastic. What an amazing opportunity and what a unique period of time to be witness to – to have lived through the birth of, and early growth of the internet, is awesome. Yet of course, we as a species have many problems and issues to address and overcome. And we can do this – if we acquire, share and use knowledge effectively.
Knowledge is the key to the future success of humanity.
In the world of digital assessment, we have an incredible opportunity – to play our part in the facilitation of the mass transfer of knowledge at a global level, but we can do an awful lot better than we are.
So for every person who is out there working in our industry to prevent innovation, stagnate new developments, inhibit experimentation and to protect their own pension at all costs, you should hang your head in shame, for you are playing your part in slowing and preventing the development of our very species.
You clowns.
CONCLUSION
In conclusion, there is a lot wrong with our industry which has barely evolved since the 1990’s. There are too man false claims, too many conflicts of interest, not enough transparency and accountability, and too many dinosaurs (Jurassic Park reference…).
In my opinion there is no excuse for so much outdated, substandard, terribly designed technology in a field of work in which this is the principal offering, and it is time for a change.
You know who you are. Where we go from here is up to you…
