This is the text of a talk I gave in London on 26 September 2019 to the UK Government Knowledge and Information Network.   I have revised and extended the text.

Think of all the correspondence moving into, out of and around your organisation.

Think of the structure or schema into which you would like all important items of business correspondence to be assigned so that they can be found and managed. Think of the records system that the structure/schema sits in.

Who would you like to file important items of correspondence into that structure/schema: humans or machines?

Trial no 1:  humans versus machines that can learn

Imagine you set up a trial:

• you tell every member of staff to file important pieces of correspondence into your records system with your preferred structure/schema;
• in parallel you set up a group of machines to look at all the correspondence coming in and out, to select important correspondence and file it into the same structure/schema as the humans.

Who would you like to win this trial-  the humans or the machines?

Who would you expect to win the trial?

Most of us in the records and information management professions would want the machines to win.  If the machines win they take the filing workload off the heads of our colleagues.  This frees our colleagues up to focus on the job they were employed for.

We would expect the machines to win provided that:

• the machines were capable of learning a fairly complex structure;
• there was a feedback loop between humans and machines so that the machines had their mistakes pointed out to them;
• the machines were learning machines that could adjust their algorithms in response to feedback;
• the trial ran long enough for the machines to improve after many iterations.

We do not yet have the automation necessary to assign correspondence routinely to a node in the kind of complex multi-level corporate wide taxonomy/fileplan/retention schedule that records managers like to use to manage records.

The nature of automation projects currently being undertaken

The type of automation projects we are seeing in information management at the time of writing are mainly based on binary questions:

• The legal world has been making progress with predictive coding projects that seek to use machine learning to answer the binary question ‘is this content likely to be responsive to a specific legal dispute?’;
• In the US NARA’s Capstone policy has motivated some US Federal Agencies to use machine learning to answer the binary question ‘is this email needed as a record?’, and a similar project is being undertaken by the Nationaal Archief of the Netherlands (their report, in dutch, is here );
• The Better Information for Better Government programme run by the UK Cabinet Office will shortly set up a project  to develop an artificial intelligence tool that can distinguish important from non-important government emails  (see the call for expressions of interest they issued in August);
• Graham MacDonald has worked on a process for using automation to support the sensitivity review of records by using machine learning to predict whether or not any particular document is likely to be covered by one of the UK’s Freedom of Information exemptions (see his thesis )

We are are going to be able to deploy machines sooner if we can find binary questions for them to resolve, than if we wait until machines can assign content to nodes within complex multi-level taxonomies/fileplans/retention schedules.  

The records management demands we make of human beings

For most of the twentieth century human beings succeeded in filing correspondence into what were often very sophisticated filing structures.   In the twenty first century this no longer holds true.   In the twentieth century humans filed correspondence because the correspondence had to be filed by humans.   In the twenty first century email correspondence has been filed automatically by the automation built into email systems.  Any injunction to civil servants asking them to move email correspondence into another system is in effect asking them to re-file that correspondence.

The automation built into email systems

The automation built into the proprietary email systems rolled out in the mid to late 1990s was not machine learning.  The machines in proprietary email systems could not learn, all they could do was follow rules.   Even now, two decades later, proprietary email systems only assign correspondence into a very simple structure and schema.

The reaction of the archives and records management community when email systems were introduced was to point out (quite rightly) the records management deficiencies of a system that aggregates correspondence into individual email accounts and does not distinguish between business correspondence and personal/trivial correspondence.  With some exceptions  (notably NARA in the US), the records and information management community has not accepted the structure of email systems as being a viable filing structure and in many administrations (including that of the UK) we have continued to ask human beings to re-file important items of correspondence into separate systems.

Trial no 2: humans versus machines that cannot learn

To go back to the idea of a trial with which I started this talk, we have for the past two decades been pitting human beings against machines:

• the humans have been asked to file important items of correspondence into a preferred records system which houses our preferred records structure/schema;
• the machines  (in the shape of email systems) have been configured to file correspondence into a simple structure that is inferior for records management purposes.

Who do you want to win this trial?   The automated filing or the human filing?

From a records/information management point of view, would you want the machines to win on the grounds that:

• they take the workload off the shoulders of our colleagues
• the filing is very predictable and consistent
• the filing is instantaneous?…..

….or would you want the humans to win because they would be filing into a structure that permits a more precise application of retention rules and access rules?

Who do you think would win such a trial?

In theory the humans have more chance of winning this second trial than they did of winning the first trial.   The human filing could prevail if the human beings in the organisation found the records structure/schema so beneficial that they would be prepared :

• to make the extra effort to file correspondence into the designated records system;
• to use the designated records system, rather than their email account, as their main source of reference for their own correspondence;
• to forego the possibility of simply relying on the inferior structure into which the email systems had filed the correspondence.

However even when officials do highly value the records structure/schema there is still a strong possibility that the machine filing will prevail.  I remember when email systems were introduced into UK government in the mid 1990s.  Government departments and the civil servants in them valued the then record systems of their organisations (hard copy registered file systems) very highly.   Everyone at the time wanted the registered file systems to survive and to make an ordered transition to the electronic world.  But within five years of the general introduction of email in UK government all of those registered file systems were in tatters with no replacement systems in place.  The introduction of email destroyed those systems.

Why did the automated filing of email systems into a simple structure overcome the value that UK civil servants placed on the much more sophisticated structure of their registered filing systems?

The crucial advantage that the machines (email systems) had was speed.  They filed correspondence instantaneously.   The automated filing by email systems provided officials with instant access to their correspondence from the moment it left the sender’s account.   This acted to accelerate the velocity of correspondence, which in turn increased the volume of items exchanged, which in turn increased the number of items to be re-filed by the human beings.

The introduction of email increased correspondence volumes exponentially and therefore made it to all intents and purposes impossible to have human beings re-file correspondence into a complex corporate structure. In other words the machines moved the goalposts.  And won the game!  

To put it more simply

• human filing is a viable option when there is a low volume and low velocity of correspondence exchange;
• if the velocity and volume of business correspondence increase exponentially then the human resource to refile it does not scale (not within public sector budgets anyway!). 

 

Machine filing versus human filing – the experience of the past twenty years

The experience of UK government in relation to email over the past twenty five years can be divided into three phases.

In the first phase (c 1995 to c 2003) human beings (civil servants) were asked to print important pieces of correspondence out and place them onto registered files whilst machines (email systems) filed correspondence into email accounts.

In the second phase (c2003 to c2010) civil servants were asked to file correspondence into electronic records and document management systems whilst machines (email systems) filed correspondence into email accounts

In the third phase civil servants were asked to file correspondence into collaborative systems (such as Microsoft’s SharePoint) whereas machines (email systems) continued to file correspondence into email accounts.

 

Over the course of this twenty to twenty five year period progress has been made in the systems to which we have been asking our colleagues to file into.  We have moved from hard copy to electronic systems; we have moved from electronic records management systems with clunky corporate fileplans to more user-friendly collaborative systems.   But the result has been the same in all three phases.   In each phase a pitifully low percentage of business correspondence has been moved from email accounts into the record system concerned.  The automated filing of email into email accounts has always defeated attempts to persuade humans to get into the habit of re-filing their important correspondence somewhere else.

The policy dilemma posed by the automated filing built into email systems

Email systems have, over the past two decades used a primitive form of rules based automation to file emails into a simple structure/schema.  This has caused a policy dilemma:

• email systems file email correspondence efficiently, routinely and predictably into email accounts BUT the organisation of correspondence into individual email accounts results in an inefficient and imprecise application of retention and access rules to correspondence;
• in contrast human beings are able to re-file important items of correspondence into a structure that enables retention and access rules to be applied more precisely BUT they are likely to do this infrequently and haphazardly.

The policy dilemma exists in part because records management best practice does not tell us which of the following two policy imperatives is more important:

• the consistent capture of correspondence into a structure/schema; OR
• a structure/schema that supports the precise application of retention and access rules.

Records management best practice does not help us choose between these two competing imperatives because records management best practice wants both!  Records management best practice requires the consistent capture of correspondence into a structure/schema that supports the precise application of retention and access rules.

We are faced with two imperfect options.  We should choose the least imperfect.  The least imperfect option is the option whose weaknesses we are most likely to be able to correct at a future date.

We are working in a period of transition, and the transition is towards the ever greater use of every more powerful automation, analytics and machine learning.  If the present rate of progress with machine learning/artificial intelligence is maintained then we can predict that:

• in the medium term originating bodies will be able to deploy machines to answer binary questions that would help to mitigate the worst faults of email accounts:  namely to distinguish important from trivial mail, and personal from business mail;
• in the long term originating organisations will be able to deploy machine intelligence to re-file correspondence into any order that they choose.

 

Factoring the future of machine learning into present-day policy decisions

If and when we reach a point at which machine learning tools can file correspondence into any order that an organisation wishes then our policy dilemma will be resolved – we will at that point be able to consistently assign correspondence to any taxonomy, records classification and/or retention schedule that an organisation chooses.  We would also, one presumes, be able to run the machine learning over legacy correspondence and assign that correspondence to the same taxonomy/records classification/retention schedule.  We can anticipate that:

• future machine learning tools will be able to retrospectively correct the weaknesses in the structure/schema of any email accounts that survive;
• future machine learning tools will only be able to retrospectively correct the weaknesses in the capture of email into corporate collaboration systems/electronic records management systems if important email accounts survive.

This logic dictates that we should give a high priority now to ensuring that historically important email accounts survive in the confident hope that we will later be able to correct weaknesses and inefficiencies in the content of these accounts and in the structure and schema of those accounts.

This would require some form of protection being introduced now for the email accounts of officials playing important roles.  Business correspondence residing in the email accounts of important UK government officials does not currently enjoy any protection.    UK government departments subject email in email accounts to some kind of scheduled deletion.  The most common form of scheduled deletion is to delete the content of email accounts shortly after an individual leaves post.  This practice complies with the National Archives’ policy towards UK government email, because each department asks its officials to move important email out of email accounts to some form of corporate records system.  However the unintended consequence of this policy is that most  business correspondence ends up being subject to this deletion.

Affording some protection to the email accounts of officials occupying important roles can be seen as a protect now- process later approach.

This protect now –  process later approach involves protecting  historically important email accounts in the knowledge that machines are good at dealing with legacy and can at a later date be deployed to filter these records, enhance the metadata and/or overlay an alternative structure on to these records.

Such an approach would no longer require individuals to move important emails to a separate system for recordkeeping purposes (though there may well continue to be circumstances when an organisation for knowledge management/operational purposes requires some teams/areas to move important correspondence out of email systems, or seeks to divert correspondence away from email into other communication channels).

This approach is based on the realisation that deploying human effort to do something (badly) that machines are likely to be able to do (well) at a later date does not make sense in terms of either effectiveness or efficiency.

GDPR implications of a protect now –  process later approach

The implication of protecting important email accounts from deletion whilst working on the development of machine learning capabilities is that some personal correspondence is likely to be retained alongside historically important correspondence.  This has data protection implications.

GDPR allows the archiving of records containing personal data provided that the preservation of the records is in the public interest, and provided that necessary safeguards are in place and the data protection rights of data subjects are respected.   The retention of the work email account of an important official is likely to be in the public interest, and is likely to be compliant with data protection law provided the following conditions are met:

• the role that the individual played was of historic interest;
• the individual could expect their account to be permanently preserved;
• the individual was given the chance to flag or remove personal correspondence;
• access to personal correspondence was prevented except in case of overriding legal need;
• items of correspondence that are primarily personal in nature are removed once a reliable capability to identify them becomes available.

 

Conclusion

This talk recommends that government departments which use email as their main channel of communication refrain from automatically deleting correspondence from the email of their  most important staff, pending the development of automated tools to process the correspondence within those accounts.   In practice this is likely to only involve protecting around 5% of their email accounts (using the old archival rule of thumb that 5% of the records of an originating body are likely to be worthy of permanent preservation).

This is not an easy sell to make to government departments.  Even though the recommendation only covers around 5% of their email accounts, departments may well feel that these are the 5% that carry the highest potential reputational/political risk, and are the 5% most likely to attract freedom of information requests. 

Making such a recommendation is in no sense ‘giving up’ on the records management ambition to have business correspondence consistently assigned to structures and schema that support the use and reuse of correspondence and that support the precise application of retention and access rules.   It is simply a recognition that asking civil servants to select and move important email into a separate system has not worked for twenty years and shows no sign of working any time soon.  It is also a recognition that we need automated tools to process the material that has been automatically filed by email systems.

Most important of all this approach of protecting important email accounts gives us a pathway for applying automated solutions to email.  It would provide an incentive and an opportunity to deploy tools that work on a binary logic (‘is this email important, yes or no?’,  ‘is this email personal, yes or no?’) to mitigate the worst flaws of email accounts from an information management point of view.  These tools are not pie in the sky, they are already being used in real-life projects.  The hope would also be that in the long term we may have tools that go beyond binary questions and could assign individual emails to a reasonably granular records classification, taxonomy and/or retention schedule.

 

 

The theories and explanations outlined in this talk have been developed during the course of my Loughborough University doctoral research project which is a realist evaluation of archival policy towards UK government email.   A paper from this project ‘The defensible deletion of government email’ was published by the Records Management Journal in March 2019.  An open access version of this paper is available from Loughborough University’s digital repository here (once in the repository click on ‘download’ to download the pdf,  or read it in the window provided).

James Lappin

 

 

 

 

 

 

At an IRMS public sector group meeting in London yesterday I heard Rob Bath compare the records retention model offered within the on-premise SharePoint with that offered in Office 365.

Rob described the two models available in on-premise SharePoint:

• the record centre model in which important items are moved to a records centre – the disadvantage of this model is that it rips content out of context by taking it from the SharePoint sites in which users had interacted with it;
• the in-place record management model in which end users can click a button to identify individual items as records – the disadvantage of this model is that SharePoint gives no reporting capability for information managers to see and manage the items scattered across their SharePoint implementation that have been declared as records.

Both the record centre model and the in-place model are still available within SharePoint Online in Office 365.  However Office 365 offers another way of managing the retention of SharePoint Online content.  This new method does not sit within SharePoint Online itself.  It sits in the Office 365  Security and Compliance centre which exists to provide a means of managing content across the Office 365 family of applications.

The Security and Compliance centre provides a facility to set up retention labels and retention policies:

• retention labels can be applied to containers within SharePoint sites such as libraries or folders;
• retention policies can be applied at SharePoint site level.

Rob’s conclusion was that the retention labels/retention policies model offered in the Office 365 Security and Compliance centre was both a simpler and more effective way of managing SharePoint content than the two models available within SharePoint itself.  One member of the audience asked him whether there were any circumstances in which he would recommend the use of either the record centre model or the in-place model in place of (or in conjunction with) the retention labels/retention policies model within Office 365.  Robert thought for a second and then said ‘no’.

In the rest of this post I will offer some thoughts on why it is that after so many years of coming up with such unwieldy retention offerings in SharePoint, Microsoft have come up with something so much better for Office 365.

The need for Office 365 to have a retention model that went beyond SharePoint

Microsoft wanted a records retention model for Office 365 that was not unique to any one particular Office 365 application, but which could be applied to all of the major applications within the Office 365 family.  This forced them to come up with a model that was not based on any features specific to SharePoint itself.  In particular it led to them to moving away from the linkage between records retention and SharePoint content types.

The need to come up with a model that could be applied to applications as diverse as SharePoint Online, the Exchange Online email system, and the OneDrive filesharing application meant that Microsoft had to look for a common denominator between the different applications.  The one common denominator between the applications is that they all aggregate content.

The records retention model in Office 365

The retention model in Office 365 is very simple.   For each application a fundamental aggregation is identified.   In Exchange it is the email account.  In SharePoint it is the site.  In OneDrive it is the OneDrive account, etc.. The Office 365 Security and Compliance centre allows you to use the fact that all content in SharePoint is held within sites, all content in Exchange is held within email accounts etc. to apply your retention rules.

The Security and Compliance Centre offers essentially two different strategies for linking retention rules to content:

• The most basic approach is that you apply retention at the level of the fundamental aggregation.   You set up retention policies in the Security and Compliance Centre and you identify which SharePoint sites (and/or which Exchange email accounts, which OneDrive accounts etc.) you wish to apply each policy to.
• A more sophisticated approach is that you manage retention at a level below the fundamental aggregation.  In this model you set up your retention rules as retention labels (rather than retention policies) in the Security and Compliance centre, and then you target the rules at the SharePoint sites/Exchange email accounts etc. in which you want them to be available for users to apply to content

Applying retention policies and/or retention labels to content in Exchange Online and in SharePoint Online 

It is possible to use different strategies to apply retention labels/policies in different ways in different Office 365 applciations,

My preference for email is to set a retention policy on email accounts based on the business value of the correspondence (which will vary according to the role played by the individual email account holder), and in addition to allow users to use a retention label to flag up personal correspondence so that it can be subject to a shorter retention period.

After his talk I asked Rob Bath whether he preferred to apply retention policies or retention labels in the SharePoint environment.  He said that he thought that a SharePoint site was typically too big an aggregation to apply one retention policy to.  He prefers to apply retention labels to libraries and folders within sites.  The way to do this is to set up your retention rules as retention labels and then identify for each label the SharePoint sites that it is relevant too.  The result will be that most SharePoint sites will only have a small number of retention labels available to manage content within them. In the process of creating any new library or new folder the library/folder can be configured to apply one of those retention labels to all content stored within it (see here).

It is important to ignore Microsoft’s pushing of retention labels as a tool for end-users to tag individual items in SharePoint.  There is no incentive for an end user to choose a retention period for an individual document.  In general it is not good practice to ask end users to do something you know they will not do.   In the SharePoint environment you should either:

• Set retention labels as default on libraries or folders; OR
• set retention policies as default on SharePoint sites (do this if setting retention labels on libraries/folders is not possible, perhaps because your SharePoint installation is too big, your number of information governance staff is too low, your roll-out schedule is too short, or you are applying retention labels/policies in a legacy environment)

 

 

Last year I gave two presentations, one to the DLM Forum Triennial and one to the IRMS conference, in which I developed a fictional case study of an organisation that decides to apply machine learning and analytics to email.

In my case study a public sector organisation:

• is concerned about the low capture of email into its record system (SharePoint) and embarks on a programme to apply machine learning to remedy the shortfall;
• uses machine learning to apply its existing policy of moving important email into a records system;
• seeks to apply the machine learning capability on all email accounts on a corporate wide basis.

In reality I think that the attitude of public sector organisations to the application of  analytics and machine learning to email will be rather different to the attitude taken by the organisation in my case study.   My predictions are that public sector organisations in the UK:

• will be reluctant to apply machine learning to email accounts because of the risks involved;
• will be just as concerned about the prospect that the application of machine learning might result in very large volumes of email being captured into their record system as they would be about the existing under-capture of emails as records;
• would use machine learning (or an analytics capability) to look for certain specific types of correspondence that are valuable to the organisation in certain specific accounts rather than applying machine learning/analytics to all accounts across the business;
• would not move emails identified as important or valuable into their corporate records system, but would instead leave them within email accounts and either place them under a hold to prevent deletion, or move them to an email archive.

Here is the video of my monologue explaining how the organisation applying machine learning to all of its email accounts got on:

 

What is an email account in Office 365?   It is a special type of document library, that doesn’t need version control, doesn’t need extra metadata fields and doesn’t live in SharePoint.

It now seems a little incongruous for organisations to ask their staff to move important emails out of email accounts and into a ‘corporate record system’.  If Office 365 is your corporate record system then email accounts are within it already!

One of the things that Microsoft had to do in order to make Office 365 work as a service offering was to get their SharePoint team working with their Exchange team – something that famously never happened whilst both products were predominantly on-premise offerings.  Microsoft customers implementing on-premise SharePoint alongside their on-premise Exchange email system  had to deploy third-party plug-ins if they wanted staff to be able to drag and drop an email into a SharePoint document library without leaving their Outlook email client.

There are two routes Microsoft could have gone with the relationship between Exchange and SharePoint within Office 365:

• the integration route – building in features that make it easier to move emails from Exchange to SharePoint;
• the governance route – making common governance features available so that emails in an email account could be governed using the same policies as documents in a document library in SharePoint.

Microsoft’s choice of direction for Office 365 has implications for the policy decisions that organisations need to take on email:

• If Microsoft were to go down the integration route then it would fit in with the records management belief that an email system is not a ‘record system’ but is instead a ‘communications tool’.  Many organisations over the course of the past decade have designated SharePoint as their corporate records system and asked staff to move important emails into SharePoint.
• If Microsoft were to go down the governance route then it would fit with the information governance belief that distinctions between record systems and non-records systems are meaningless and unhelpful because organisations are under legal, regulatory and ethical obligations to manage all their business information systems in accordance with information governance principles.

From a marketing point of view, there are clear advantages to Microsoft from going down the governance route rather than the integration route:

• If Microsoft went down the integration route it would imply that they viewed a SharePoint document library as a better place to store business email than an Exchange email account. This is despite the fact that Exchange was built for and designed around the storage of email messages, whereas SharePoint document libraries were not designed with email in mind.
• By going down the governance route Microsoft can stay neutral on the question of whether an email is better stored in SharePoint or in Exchange, and can gradually remove any necessity for organisations to move emails out of Exchange and into SharePoint.

It is therefore no surprise to see Microsoft putting their emphasis on the governance route rather than the integration route.

Office 365 comes with a ‘Security & Compliance Centre’ that sits separately from SharePoint or Exchange or any of the other component parts of Office 365.   The Security & Compliance Centre gives you two different means of applying retention rules to content:

• retention policies which are applied to the containers within which content sits (SharePoint sites, email accounts etc.);
• retention labels which are applied to individual items of content (emails/documents etc.).

This effectively gives you three alternative options for applying retention to email:

• apply retention policies to email accounts without applying retention labels; OR
• ask end users to apply retention labels to emails (or automate the application of labels if and when you develop automation capability), without applying retention policies; OR
• use a combination approach by applying a default retention policy to email accounts whilst allowing staff (or machines!) to apply a retention label to particular emails that deserve a retention rule that differs from the default.

Note that in applying retention from the Security & Compliance Centre to content in OneDrive, Office 365 groups or SharePoint you will be faced with variations of the three options listed above.   The variation relates to the type of container that you would be applying retention policies to, and the type of content that you would be applying retention labels to.

The fact that Microsoft allows an email account to be treated in the same way as a document library for retention purposes will not stop organisations wanting to apply different retention periods to email accounts than to document libraries even when they arise from a similar business function.  The cost and risk profile of an email account differs significantly from that of a document library.

However Office 365 is a game changer in two ways:

• it brings the application of retention rules to email in email accounts firmly into the information governance, rather than the IT domain.  The retention policy  and retention label menus in the Office 365 Security & Compliance centre can be used to apply retention policies and/or retention labels to Exchange email accounts and SharePoint sites (as well as other parts of Office 365 including Office 365 groups and OneDrive accounts);
• it creates the possibility of applying different types of policy towards email. For example if you wanted to apply a Capstone policy towards email you could do so out of the box in Office 365 by simply:
  • setting two retention policies on email; a Capstone retention policy for application to the relatively small number of email accounts that you wish to retain permanently, and a non Capstone retention policy for application to email accounts that you do not wish to retain permanently;
  • deploying  retention labels to enable staff with Capstone email accounts to identify trivial and personal emails so that those emails are exempt from the permanent retention applied to the rest of the correspondence in their email account.

I will be giving a presentation in London on Friday March 15 2019 for the IRMS Public Sector Group.

Below is a summary of the presentation:

James Lappin has recently had an article published by the Records Management Journal The defensible deletion of government email in which he reports on the evaluation he has carried out with Loughborough University of the policy of The National Archives (TNA) towards UK government email.

In this presentation he will attempt to answer three questions:

• what proportion of an official’s email correspondence is likely to be needed as a record?
• what proportion of an official’s email correspondence are public sector bodies likely to be able to capture into their record systems?
• what proportion of an official’s email correspondence are public sector bodies likely to want to capture into their record systems?

On the basis of the answers to the questions above,  James will propose an answer to two further questions:

• why has no solution to the email problem yet been found, after nearly a quarter century of its dominance of business correspondence?
• is it likely that a solution to the email problem will be found that will be acceptable to both public sector bodies and to wider society?

The basic premise of the talk is as follows:

• if the proportion of email correspondence that is needed as a record exceeds the proportion of correspondence that public sector bodies are capable of capturing into their records systems, then we as a records management/information governance profession have a problem that we can and will solve, because technical solutions that we identify will be welcome to our organisations;

 

• if the proportion of email correspondence that is needed as a record exceeds the proportion of correspondence that a public sector body perceives as being in their interests to treat as records, then we have a problem that we cannot solve, because any technical solution we could come up with would be unwelcome to our organisations, and against their interests to adopt.