Data comes in many forms, and machine learning has the potential to turn all of it into predictive, actionable insights. Here’s how it works, and why it matters.

Business intelligence, commonly referred to as BI, is a powerful tool for organisations. By crunching the data available in core business systems, BI can empower everyone, from the C-suite on down to workforces servicing customers on the front lines, with visualised and relevant insights that empower real-time decisions – and drive business outcomes.

Research-backed prediction tools are likewise valuable. For example, behavioural science research-backed assessment and intervention tools have a successful track record of leveraging behavioural science research and data to support progress and outcomes for job seekers in the employment services space. Do we really need more than these tools?

There’s one exciting innovation that goes even further than both. Machine learning.

What is machine learning (and why does it matter)?

Machine learning is a branch of artificial intelligence and computer science that uses data and algorithms to imitate the way humans learn. ML algorithms create models based on sample data to make their own predictions or decisions. They can also gradually improve their accuracy over time, as new data becomes available and informs future predictions.

Machine learning has a lot of real-world applications. For example, it’s currently being used for everything from image and speech recognition, to medical diagnosis, fraud detection, Google language translations, and the social media platforms we use every day. The likes of Facebook, Spotify or Coursera are experts at using machine learning to serve up content.

Machine learning has a lot of future potential for Australian businesses, including our education, skills and workforce sectors. Because when it comes to the full (and increasingly flexible) human journey taken through learning and work, there’s a lot we can learn from the abundance of data that we are creating to improve experiences and outcomes.

The potential of machine learning at ReadyTech

ReadyTech is the custodian of lots of data. This data has been collected over a long time in our systems for employment services, apprenticeships, training providers and others. These include highly structured data sets that are well labelled and easy to access (with common IDs to link to external data sets) as well as treasure troves of untapped unstructured data.

This makes us a veritable playground for exploring the potential of ML. Using the profound potential of machine learning neural networks, we are now able to use both captured structured and unstructured data to predict future outcomes with an extremely high level of accuracy. Where BI is exceptional in seeing the past, ML is now seeing into the future.

• Apprenticeships sector

The true impact of ML is this combination of structured and unstructured data to create a fuller, more predictive capability for the users we serve. In the apprenticeships space, this might mean combining data points like date of commencement and completion (structured data), with mentoring notes (unstructured data) to make more accurate future predictions.

• Employment services

In employment services, job seekers and their outcomes are present in the data, and so is the effort exerted by providers to arrive at outcomes, both good and bad. It is possible for us to capture, log and analyse user interactions, and our proximity to the process allows us to operationalise insights more readily in systems, meaning time to value is reduced.

How does machine learning actually work?

What if you knew with between 85% to 95% accuracy which apprentices or job seekers could benefit from which types of support and interventions at different stages of the service lifecycle, as well as those who did not need these resources and were fine? That’s the promise and potential of data when machine learning neural networks are applied.

Traditional programming is a case of plugging in the rules and the data and spitting answers out the other end. But in some ML use cases, we already have the answers. What we want from machine learning is to plug the answers and data into an algorithm or neural network, which does the necessary ‘thinking’ required to come up with the rules themselves.

Take the example of a job seeker in employment services. We have the data which tells us what outcomes have been achieved. Through machine learning, we can establish the rules for achieving employment outcomes, and create a risk rating which identifies how at risk an individual is of not making progress towards – or indeed achieving – employment.

ML achieves this using neural networks. Rather than linear regression - which predicts a line of best fit in two dimensions - machine learning learns in multiple dimensions. It can be understood as linear regression in multi-dimensional space. For example, the model we’re working in apprenticeships to predict outcomes works on 1040 forward dimensions.

For unstructured data, ML models use vectors. Feature vectors take data and organise it by representing numerical or semantic characteristics in a mathematical, easily analysable format. For example, in mentor notes, vectors can help to structure and analyse natural language to identify semantic meaning, and create clusters based on patterns in data.

This allows machine learning in markets like apprenticeships, employment services or education to group and split individuals off into different risk profiles, based on different characteristics. ML is responsive to new data as it is fed into the system, and also allows us to make any outputs ‘explainable’, so we know how a risk rating is arrived at.

Machines can learn (and teach us) a lot

Exploring the potential of machine learning requires technologists to take a cautious approach. For example, ReadyTech has taken pains to ensure our models and predictive indicators are 'explainable', as we are dealing with the lives of individuals in unemployment or training. We also need to be careful about what data is utilised, including any of a sensitive nature.

It is absolutely critical that we are able to get this right, for our customers and their users.

But one thing is clear. ML is beginning to reveal its predictive power and value.

The unstructured data component is particularly interesting. Going beyond BI, it means every time a consultant or administrator in employment services, apprenticeships or education enters notes into their system, they are actually contributing to continuous improvement the future. They are actively delivering the capacity for better work, training and employment outcomes.

When we see the benefits for Australia’s job seekers, apprentices or students, the argument for technology augmenting our approach becomes very strong. Technology and machine learning has an important role to play. And the time to get excited about that impact is now.