In order to manage your organisation’s digital heap you will need to make numerous interventions, at different times, in different repositories, for different purposes, on content that is at different stages of the records lifecycle. You are therefore going to need a tool kit of complementary methods and approaches.

For any particular  intervention, you have two key tactical choices to make. One comes from records management, the other comes from data science.

The key records management choice

The key records management choice is the level of aggregation at which you intervene. Corporate collaboration/document management systems and corporate email/messaging systems all have a structure. Teams and individuals create and receive content within a container (site/drive/account) that their organisation has provisioned to them.

Any intervention on any particular repository can either:

• be targeted directly at all individual items across a repository OR
• act on and within containers

The existence of a choice of levels of aggregation at which to act means that there is always a way of applying records retention and/or sensitivity categories within a repository.

If it is not feasible to act on individual items across a whole repository, and if acting on containers is not precise enough, then you can first act on all containers across the repository, and later intervene within a subset of those containers. The intervention could re-assign items that don’t fit the category assigned to their container.

The key data science choice

The key tactical choice in data science is between supervised and unsupervised learning approaches.

Supervised learning – training models to classify content

In supervised learning you build models. You teach an AI model what categories you want to apply to the target datapoints, and then test that it is applying them correctly. The classic mechanism of supervised learning is the classifier, which learns two or more categories and assigns each data point in the target data set to one of those categories. In a records management context you might seek to apply retention or sensitivity categories to content.

In data science the big strength of supervised learning is the ability to leverage and scale existing knowledge. You transfer your knowledge to the model, which applies it over a larger scale than humans could reach.

Unsupervised learning – using dashboards to understand content

In unsupervised learning you use AI to visualise a set of data. These methods teach you about the data. They surface features, trends, structure, relationships and/or groupings within the data – and you can then use that as a basis for making decisions on the data.  Examples of unsupervised approaches include:

• analytics – surfacing of summary statistics about a dataset, or about groupings within a dataset
• summarisation – textual summaries of a dataset, of groupings within a dataset, or of items within a dataset
• entity extraction – extraction of the names of people, places, organisations and/or projects/programmes mentioned in a dataset or a subset of a dataset
• cluster analysis –  assignment of each datapoint/item to a grouping in such a way as to maximise the similarity of all the items in each group

These methods can be brought together into a dashboard for a records team to analyse containers (drives/sites/accounts) and intervene within them.

In data science the big strength of visualisation/unsupervised learning is that it can generate insight, learning and new knowledge. If a retail or social media company wants to understand their customer base then running a cluster analysis of the buying or browsing behaviour of end users is a great way of doing it. The clustering might throw up unexpected similarities between particular types of customers/users. In a records management context if you want to understand the contents of an email account then running a cluster analysis within the email account is a good way of quickly seeing the main areas of activity that the individual was involved in.

The complementary nature of supervised and unsupervised learning

Unsupervised and supervised methods are complementary. They have different strengths and work well in different circumstances.

Supervised learning works well:

• when the categories you wish to apply have clear boundaries
• when one way of categorising the data is clearly superior to all others (or is mandated by regulation/law/policy commitment/ contractual commitment)

Unsupervised learning works well:

• when you first need to understand the data before applying labels/categories
• when the categories that you wish to apply have fuzzy boundaries (and hence it is difficult to teach a model how to apply them consistently)

The two approaches can be used in tandem. For example you can use unsupervised learning to increase your knowledge of a dataset, and later cement that knowledge in the form of a model that can classify future datapoints/items/containers.

A data scientist will need to intervene on different types of data sets, for different purposes, at different times. The fact that these two complementary sets of methods exist means that there is always a way that a data scientist can act on a data set to derive insight or to make the data itself more manageable.

Three viable options

When you combine the key records management choice with the key data science choice you get four options. One of the four options is not viable in a records management context. It is not feasible to use unsupervised learning to inform actions on individual items all across a repository.  That still leaves you with three viable options. You could:

• use supervised learning to apply categories to individual items
• use supervised learning to apply categories to containers
• use unsupervised learning to inform action on containers and to inform interventions within them

This means that there always exists a viable option for you to carry out a records management intervention using data science methods in any corporate collaboration/messaging/document management system at any stage of the records lifecycle.

We can predict that each of these three options will be needed at one stage or another of the records lifecycle. We can also predict that no single one of these three options will be able to fuflil all records management purposes, in all repositories, at all stages of the lifecycle.

A key element of the building of a data science specific to the domain of records management will be capturing insights as to the trade offs involved in using these different approaches for particular records management purposes, and of the circumstances in which they work well/not so well.