How the Cabinet Office’s 90 day email deletion was reported back in 2004

The story that the Financial Times [1] and other newspapers ran last week about the Cabinet Office 90 day email deletion practice is not new.   It was reported on 18 December 2004 by the BBC, two days before the deletions were about to start.   Here is the first part of their report [2]

Screen Shot 2015-06-20 at 13.51.04

I tweeted a link to the BBC report on Saturday .   I received this tweet in reply

Screen Shot 2015-06-20 at 14.10.02

Good question!

Imagine yourself back on the 18 December 2004, a fortnight before the UK’s Freedom of Information Act comes into force. It comes to light that the Cabinet Office, at the very heart of Government, is planning to have all emails that are over 90 days old deleted from their email servers.    In other words they are deleting about ten years worth of correspondence – a portion of which will have been captured elsewhere, but much of which will only have existed on those email servers.   The opposition leader has raised objections.   Why did archivists and records managers not raise this as an issue?

There are four main reasons why archivists/records managers did not object at the time:

  • We had spent the previous ten years warning our organisations about the inconveniences and dangers of records building up in individual email accounts.  We therefore were not minded to defend those records that had built up in Cabinet Office email accounts
  • We did not know what to do with email accounts.  We did not know how we would manage them over time,  how we would deal with the personal data within them, how we would sensitivity review them, what retention rules we could apply to them, nor how we would appraise them as being worthy or unworthy of permanent preservation.  We could not envisage how or when we could make available to the public an email account selected for permanent preservation.
  • Records management in the UK  had  a great year in 2004,  mainly thanks to the Blair government.  During 2004 central  government threw money at electronic records management systems, created lots of new records management posts, employed lots of records management consultants.  It was exciting.  We were not in a mood to question things too closely.
  • On December 18 2004 we were all so focused on the coming of FOI (and perhaps the coming of the Christmas holiday) that we didn’t notice.   I don’t remember even seeing this article, I can’t remember anyone mentioning it to me.

This is an important case with significant implications for records management and archival practice.   We might usefully debate the following questions:

  • Was the mass deletion of  Cabinet Office emails from email servers in December 2004 carried out for political advantage, administrative expediency or for recordkeeping improvement?   Has the Cabinet Office continued that  90 day auto-deletion policy under successive Labour, Coalition and Conservative administrations for political, administrative or recordkeeping reasons?
  • How serious is the impact of these deletions on the historical record?  Is the important correspondence largely captured elsewhere? Or is this 90 day auto-deletion of email by one of the most important Whitehall Departments going to create a significant and irreplaceable gap in our nation’s historical record?
  • Are we as a profession – records managers and archivists – any better equipped to manage email accounts over time now than we were back in 2004?  Is there a feasible alternative to auto-deletion?
  • Should the UK National Archives follow the example of its counterpart in the US and step in to prevent the auto-deletion of significant email accounts by declaring that it requires UK Government departments to select the email accounts of  important civil servants for permanent preservation?


[1] Pickard, Jim and Stacey, Kiran  16 June , 2015 8:17 pm  Freedom of information is Mission Impossible for Downing St emails.   Financial Times, ,   available from (it requires a log-in).  Accessed 17 June 2015    OR see see Morris, Nigel  17 June 2015.  Government faces call to review self destruct email policy (accessed 20 June 2015).

[2]  BBC News, 18 December 2004, 14:56 GMT.   Howard condemns email deletion. Available at  (accessed 20 June 2015).

Thoughts on the UK Cabinet Office’s 90 day auto-deletion of emails

The Financial Times [1] reported this week that the Cabinet Office auto-deletes their e-mails after 90 days.

In the article several officials and aides are quoted describing day to day problems that Cabinet Office staff have experienced in the absence of an adequate correspondence record:

  • A special advisor is quoted as saying that ‘It means that people don’t remember things…It is dysfunctional.  Then they check their emails and they don’t exist anymore’.
  • An aide is quoting as expressing their frustration that colleagues often had different recollections of what had been agreed at meetings.

The Cabinet Office’s  90 day deletion policy was introduced in 2004.   The Freedom of Information Act for England and Wales came into force on the 1 January 2005.    The article quotes  Maurice Frankel of the Campaign for Freedom of Information  as stating that the proximity of those two dates ‘was not a co-incidence’.   However the article  also reports: 

An official said the system had been recommended by the National Archives for  ‘best records management’ and it was a coincidence that it began at the same time as Freedom of Information

On the one hand I do not believe that the Cabinet Office’s 90 day deletion policy was introduced in 2004 purely as a conspiracy:

  • In 2004 many Whitehall departments were introducing electronic records management (ERM) systems, into which they told staff to capture significant documents and correspondence.   
  • It is quite possible that the Cabinet Office’s routine deletion policy dates from the time of their roll out of their ERM system
  • The UK National Archives advice was at the time (and still is now) [2] that Departments should capture significant correspondence into a designated records system and should routinely delete correspondence from e-mail accounts.
  • As FOI man [3] points out,  back in 2004 most records managers would have recommended that an organisation implementing an ERM system  should also routinely delete e-mail left in email accounts
  • At the time, in 2004, we as archivists and records managers genuinely hoped and believed that the policy of requiring colleagues to declare important emails and documents into an ERM system would capture a reasonable record of their correspondence
  • The ERM systems introduced around 2004 by many UK government departments, including the Cabinet Office, had been tested and certified by the UK National Archives as meeting their specification of requirements for the management of electronic records
  • The UK government was the first administration in the English speaking world to implement ERM systems on a large scale.  If felt like an experiment at the time – powered by the Blair government’s modernising government agenda, the coming of FOI in England and Wales (and separately in Scotland), and by the fact that paper records systems had been blown apart by the coming of email in the mid 1990s.

On the other hand :

  • just because archivists and records managers in the UK government sector and elsewhere  recommended the routine deletion of email from email accounts in good faith,  back in 2004, whilst embarking out on the electronic records management system journey ,  and in the hope and on the assumption that the policy instruction to save significant emails into electronic records management systems would be followed….
  • …..does not mean that we should continue to recommend or support such routine deletions now, in 2015,  in the light of the experience that we have gained in the intervening decade.

Experience has taught us that asking users to declare important emails as records into any sort of record system does not result in the capture of an adequate correspondence record

The experience we have had in the UK, in continental Europe, in Australia and in the US is that asking colleagues to choose which of their e-mails are significant, declare them as a record, and move them to some sort of electronic file (or paper file, or SharePoint document library) does not capture an even vaguely adequate record of correspondence.    

It is interesting to read the quotes of the special advisors/aides/officials in the FT article.   One presumes that some emails have been captured into the Cabinet Office’s ERM system (for nearly a decade they had a system provided by Meridio, but have recently replaced it with Google Docs).   And yet the officials quoted by the FT  talk as though there was no correspondence record at all. My guess is that this is because individuals saved so little correspondence into the system,  and saved into the system so selectively, that to all intents and purposes there was and is no Cabinet correspondence record outside of e-mail accounts.

This is only a guess – we would need to know the figures.  The most relevant figures being:

  • how many e-mails did Cabinet Office staff send in any given year?
  • how many e-mails were captured into the Cabinet Office record system in that same year?

We do have these figures for the US State Department, thanks to a report [4] published by the Office of Inspector General in March 2015.   The report states that in 2009 the State Department implemented a record system that was built into the Outlook email client, whereby individuals could declare important emails as records.   This was backed up by a policy instruction that individuals were responsible for preserving important emails as records, and by training.

The Office of Inspector General’s report sates that in 2011 the State Department sent 1 billion emails, but only 61,556 were declared as records.   Robert Smallwood [5] calculates that the emails captured into the State Department’s  record system constitute only .0061%  of their total email communications.  In 2013 the situation was even worse – only 41,749 State Department emails were declared as records.   Interestingly the report states that one of the reasons for the lack of the declaration of emails as records was a fear that they would show up in searches (which I am interpreting to mean that they would show up in searches conducted in response to Freedom of Information requests).

If the Cabinet Office no longer knows how many emails it received in a given year then the headcount of the Cabinet Office in that year would be the next best figure.    Chris Prom [6] quotes statistics from Radicati [7] stating that on average a business user sends 33 emails a day.

I recently spoke to an organisation who had a reasonably good electronic records management system, no fear of Freedom of Information, and a strong commitment to record keeping.  They told me that the number of items contributed to their records system equated to one item per member of staff every two days.  If the figures Prom quotes are correct that would give a ration of  around 1 email saved to their records system for every emails 66 sent.  That is several orders of magnitutude better than the State Department’s ratio, but for an important organisation that ratio is still nowhere near high enough to function as a useable and defensible correspondence record.

Most UK government bodies operate routine deletion policies, albeit significantly less drastic than the Cabinet Office’s 90 day deletion.    Rather than a 90 day deletion it is more common for the deletion to occur  1 or 2 years after the email is sent, or six months to a year after the individual email account holder has left employment.  Alternatively or additionally some departments operate a limit on email box size rather than an automatic deletion.  

Individuals in such departments do not experience the inconveniences reported by Cabinet Office staff.   They are cushioned from the impact of the deletions, but the net result is the same.  The correspondence record being kept is inadequate and cannot support basic succession planning/staff handover in the short term, corporate memory in the medium term and historical accountability in the long term.


[1] Pickard, Jim and Stacey, Kiran   2015.   Freedom of information is Mission Impossible for Downing St emails.   Financial Times, June 16, 2015 8:17 pm,   available from (it requires a log-in).  Accessed 17 June 2015

[2] The National Archives (TNA) (date unknown) Managing emails.   Available from  (Accessed 18 June 2015).    TNA’s policy advice to Government departments is  that

”You will need to…..limit what users can keep in personal email accounts by the use of:
– email account quotas OR
– automatic deletion after a set period of time”

[3] Gibbons Paul (FOI man) 17 June 2015.  FOI avoidance or good records management?  Cabinet Office Email Policy.  Available from  accessed 18 June 2015

[4] Office of Inspector General, March 2015. Review of State Messaging and Archive Retrieval Toolset and Record Email (ISP-I-15-15) Available from or access the pdf directly from

[5] Smallwood, R, 2015.  Clinton Email Kerfuffle: Call for Cabinet-level Information Governance.  Available at 17 June 2015)

[6] Prom, Christopher J, 2011.  Preserving email (Digital Preservation Coalition Technology Watch Report)  pdf available from  On page three of the report Prom writes ‘Over 3.1 billion email accounts currently exist, and the average business user sends 33 email messages per day (Radicati Group, Inc. 2011a). ‘

[7] Radicati Group, Inc., 2011a. Email Statistics Report, 2011–2015, Executive Summary. Available at: [Accessed July 17, 2011]. (Quoted by Prom)

Records management is wanted – dead or alive

I attended the Information Governance and eDiscovery summit in London last week.

Two friends,  both  stalwarts of the records management world in the UK, separately came up to me and said ‘you know what I think Records management is dead now James’

Records management isn’t dead.  It is a contestable space at the heart of information governance.

In the e-Discovery stream of the summit there was no mention of any type of document management or records  system (whether that be an electronic records management system, or SharePoint).  This is because these systems are often not relevant to the preparation of lawsuits, where the lawyers go straight to the e-mail record and/or text messages to establish the chain of events and discussions pertinent to the case.

In effect organisations are operating two parallel spheres of recordkeeping.

  • the communications/correspondence sphere that consists mainly of their e-mail servers/e-mail archive,   but also any capture of instant messaging and/or text messages
  • the document sphere that consists of whatever combination of electronic records management (ERM) system/SharePoint/line of business system/fileshare applications they deploy

The key difference between the correspondence/communication sphere and the document sphere is that:

  • an e-mail server/archive can be set up to capture all the traffic going to and from a particular e-mail account on a particular server.  Similar arrangements can be set up for text messages on company devices and for instant messaging systems.
  • an organisation wide ERM system/SharePoint implementation captures only those documents/communications that an individual chooses to declare to that application.

The paradoxical relationship between the two spheres of recordkeeping

This split between e-mail servers/archives on the one hand and corporate ERM systems/SharePoint implementations on the other creates the following paradoxes:

  • Organisations fear the contents of their e-mail archive/servers but lack confidence in the completeness of their designated record repository (ERM/SharePoint etc.)
  • An implementation of an ERM system/SharePoint or similar has an advantage over an e-mail server/archive in that  the information in it poses a lower risk to the organisation.  Sensitive personal information is likely to be confined to predictable places within the system, and unguarded comments are less likely to find their way onto the system.      
  • an e-mail server/archive  is more effective than an ERM system/SharePoint in enabling an organisation to account for itself in difficult or controversial circumstances.  For example in situations such as litigation, investigation, or hostile freedom of information requests.   However an e-mail server/archive is not useful for non- contentious, day to day internal usage, because the undifferentiated presence of sensitive personal communications means we cannot allow colleagues to access/search each other’s email accounts.
  • An ERM system/SharePoint is good for non-contentious day to day internal usage but is weak in times of dispute, litigation and investigation.
  • Some organisations configure their e-mail server/archive so that it retains a copy of every message even after an individual has deleted that message from their e-mail client.   This best ensures they have have a complete record of e-mail correspondence and thus best supports accountability.  However it also means that they are more likely to hold unguarded, trivial and sensitive personal e-mails because the individual account holders will not be able to delete them
  • The weakness of corporate ERM systems/SharePoint  implementations is that they are incomplete.  They do not contain relevant documents or correspondence if the individuals who handled them lack the time, motivation and/or awareness to put them onto the system.
  • Organisations are operating two separate sets of governance arrangements for these spheres    Records managers might typically set  the retention rules for the ERM/SharePoint sphere.    IT managers and/or legal managers might set the rules for the retention of communications/correspondence stored in e-mail accounts and for instant messages and text messages.
  • Most records management retention schedules are written as though they are universal and format neutral.   They often make provision for relatively long retention periods for certain types of correspondence (ministerial correspondence, correspondence with regulators, correspondence with foreign powers etc.).   And yet these retention schedules are rarely applied to the e-mail accounts which hold most of that correspondence, and which are often deleted after a relatively short time period.   Note that NARA’s recently issued retention schedule for e-mail accounts of US Federal agencies is an exception to this rule (NARA, 2015)

The task of information governance

Information governance is often described as an umbrella term covering the separate disciplines of eDiscovery, records management, access to information and privacy.    

The inclusion of both eDiscovery and records management under this umbrella means that information governance spans both:

  • the communications/correspondence sphere (e-mail archives/servers and any other communications tools deployed)  AND
  • the document management sphere (ERM systems, SharePoint and other collaboration systems,  shared drives)

One of two things could happen here.

One possibility is that each of the component disciplines of information governance stay entirely separate.   Records management carries on asking people to move important correspondence to an ERM/SharePoint or whatever tool they designate as their main repository.   Lawyers perfect their eDiscovery/analytics tools which allow them (and only them) to search across the email server/archive and across all repositories in the organisation.

The other possibility is that information governance acts to influence these disciplines, to create synergies and to enable them to become more than the sum of their parts.  To an extent this is happening already as analytics tools honed on eDiscovery cases cross over into the records management space to enable organisations to apply disposition decisions to hitherto intractable repositories such as shared drives.

Information Governance should work towards becoming the arena in which the tensions and contradictions between accountability, risk, and privacy can be resolved or managed in relation to both the main correspondence system (which at the time of writing happens to be e-mail) and the main document management applications and repositories.

The task of records management

I would like to see records management use information governance to blur the  boundaries between the systems they designate as records systems, and the other repositories in the organisation including, but not limited to, e-mail servers/archives.    Lawyers use their eDiscovery tools to search across all repositories in the organisations.     As records managers we should be seeking to stretch their retention remit to cover all repositories.  We should be seeking to establish relationships between content held in whatever we designate as our main records repository and related content held on shared drives, in e-mail accounts etc.

Cutting through the e-mail paradoxes

Organisations would like to be in a position where they can dispense with the complete record of e-mail communications after as small a time interval as possible, and rely instead on the  filtered, lower risk records that exists in their document management sphere.   However the swift deletion of e-mails from their email server/archive adversely impacts their ability to account for their actions.

The irony is that external stakeholders/litigants/regulators/hostile FOI requesters are not  interested in trivial e-mails, and they are not interested in personal e-mails. External stakeholders would be happy for organisations to delete or redact them.

The problem is that organisations cannot currently defend or explain how they got from the complete correspondence record on the e-mail server/archive, to the filtered record on a corporate electronic records management system/SharePoint/shared drive.

Records management should work towards providing organisations with a defensible consistent, routine and transparent way of distinguishing trivial, sensitive and personal e-mails from typical business correspondence, with a view to enabling organisations to have a record of each individual’s business correspondence that is both filtered and comprehensive;  defensible and accessible; to which defensible retention rules can be applied, and which can be linked to related collections of documents and communications  held in other applications/repositories, including the designated records repository (ERM/SharePoint or similar).


NARA (2015),   GRS 6.1 Email Managed Under a Capstone Approach (accessed 19 May 2015)

Ravanbakhsh, Arian (2015) DRAFT Capstone GRS Available (accesssed 20 May 2015)

An attorney advises clients against the routine deletion of e-mail

The view that a legal team takes on how long emails/ email accounts  should be retained varies from organisation to organisation.  Here are two examples I have come across in the past couple of years:

  • In one organisation the legal team wanted the e-mails in the e-mail archive kept indefinitely because whenever there was any legal dispute it was the e-mail archive that they relied upon to build their case.  The records manager was concerned about this on the grounds that a significant proportion of the e-mail was trivial and/or personal.    The records manager asked for a two year retention period.  In the end they compromised and went for a seven year period.
  • In one organisation the legal team want the organisation to destroy e-mail after a year in order to reduce the chances of them having to disclose them in response to Freedom of Information requests.  The records managers support the legal team because they expect that the deletions will lead to more correspondence going into the electronic records management system

So which legal team is correct?  And should records managers be pressing for short or long retention periods on e-mail?

From a lawyer’s point of view it boils down to a simple equation:

  • Does the value to an organisation of keeping an e-mail account in terms of helping them to account for their actions and in helping them prepare their case to prosecute or defend a lawsuit outweigh the risk of being obliged to make damaging revelations to litigation opponents or in response to freedom of information requests?

Ralph Losey publishes the e-Discovery team blog.  He is over in London this week for the Information Governance and e-Discovery summit.   He is Chair of the Electronic Discovery Practice Group for the US law firm Jackson Lewis P.C.

Jackson Lewis is a firm that specialises in employment law.   Ralph is unequivocal that he wants his clients to retain their e-mail.  He told me that most of the employment claims his company defends against are spurious, and that it is easier for him to prove that the claim is spurious if his client has retained their e-mail correspondence from the period in question.

Ralph pointed out that most employees in most companies act legally.  The e-mail record is going to defend them more often than it will incriminate them.

This is what he said about email deletion in this recent  blogpost (Losey, 2015)

My understanding and experiences with Big Data analytics over the last few years have led me to understand that more data can mean more intelligence, that it does not necessarily mean more trouble and expense. I understand that more and bigger data has its own unique values, so long as it can be analyzed and searched effectively.

This change of position was reinforced by my observing many litigated cases where companies no longer had the documents they needed to prove their case. The documents had short retention spans. They had all been destroyed in the normal course of business before litigation was ever anticipated. I have seen first hand that yesterday’s trash can be tomorrow’s treasure. I will not even go into the other kind of problems that very short retention policies can place upon a company to immediately implement a lit-hold. The time pressures to get a hold in place can be enormous and thus errors become more likely.

There is a definite dark side to data destruction that many do not like to face. No one knows for sure when data has lost its value. The meaningless email of yesterday about lunch at a certain restaurant could well have a surprise value in the future. For instance, a time-line of what happened when, and to whom, is sometimes an important issue in litigation. These stupid lunch emails could help prove where a witness was and when. They could show that a witness was at lunch, out of the office, and not at a meeting as someone else alleges.

For what its worth I think we as records managers should press for e-mails to be kept for as long as we would keep the correspondence of the individual role holder if they kept a correspondence file.


Losey, 2005, Information Governance v Search: The Battle Lines Are Redrawn, e-Discovery team blog, , 8 February 2005 [accessed 13 May 2015]

Managing e-mail in its native environment

The main differences between the e-mail policy of the US National Archives (NARA) and those of the national archives of Australia, Canada and the UK, are that:

  • NARA would rather accession and permanently preserve the contents of the e-mail accounts of senior federal civil servants than have those e-mail accounts routinely deleted.
  •  NARA does not insist US federal agencies move significant correspondence out of their e-mail environment into a separate records system

This is an important development.   Not because it solves the challenge of e-mail.  It doesn’t.  E-mail accounts are still hard to manage because of the undifferentiated presence of sensitive personal data about the account holder and/or the people they correspondence  with and/or third parties.

It is important because it gives a green light for  archivists and records managers to explore ways of managing e-mail correspondence within an e-mail environment.

Over the past fifteen years the main ambition of records management practice has been to move significant e-mails out of the e-mail environment (typically Microsoft Outlook/Exchange) into a separate  ‘record system’ ( paper files/shared drives/electronic records management systems/SharePoint etc.).

The problem with this approach is that an e-mail environment is optimised for people to navigate, browse, sort and read e-mail.   In contrast a document environment such as  SharePoint is not optimised for e-mail.

SharePoint is a system that is designed to be so flexible that an organisation could, if it so wished,  define a different set of metadata columns for every document library in every different SharePoint site.     E-mail on the other hand has a fixed set of metadata fields that are common to every e-mail.

We are accustomed to browsing e-mail in a completely different way to browsing documents.  We browse e-mails by date, sender or recipient.  We browse documents by activity or subject.

Basing the practice of an entire profession (records management) on moving content (e-mail) from an environment that  is optimised for it to an  environment that is not optimised for it is not a recipe for long term success.   

As a profession and as individual practitioners we cannot change this approach overnight.  But we can start to explore what policy provisions we would need, and what alterations/additions to e-mail environments and their ecosytsems we would need,  in order for e-mails to be manageable over time, and shareable over time, within an e-mail environment.


See this previous post for extracts from the email policies of the four national archives mentioned in this post

Policy advice on e-mail from different national archives

This blogpost consists of extracts from the advice on managing e-mail that the national archives of Australia, the UK, Canada and the US currently provide to the agencies/departments/ministries of their respective governments.

I am posting this simply to record the difference in policy between the national archives  of the US  (NARA) on the one hand and those of the UK, Canada and Australia on the other.   My next blogpost will give some background to the difference and on the dilemma that national archives face in relation to e-mail.

The four extracts were taken from policies accessed from the websites of the respective archives on 30 April 2015.

The National Archives of Australia

The following extract is taken from ‘Managing e-mail‘  available from the National Archives of Australia website. The navigation pathway is Home/ Records management /Managing your agency recordsDigital recordsManaging email

”You should store business email in a system that can manage it effectively for as long as it is needed. This could be an electronic document and records management system (EDRMS), a case management system or another suitable business system.

If your organisation doesn’t have a more suitable system, it’s better to store your business email in a network or shared drive system than leave it in an email system. Your information will at least be available to other staff to use and it can be stored in context. However, information in shared drives can be altered or deleted without authorisation so this should only be a temporary solution.”

The UK National Archives (TNA)

The following extract is taken from the ‘Managing e-mails’ page of the TNA website,.  The navigation path is Home > Information management > How to manage your information > Policy and process > Managing emails

Emails are an important part of the corporate record for all organisations. For public sector bodies they are public records and are subject to the Public Records Act, the Data Protection Act and the Freedom of Information Act. Therefore they need to be managed in a way that meets legislative requirements.

All civil servants have an obligation to keep accurate official records under the Civil Service Code.You will need to:

  • define clearly which emails need to be kept for business or historical value

  • communicate simply and often to users the rules for what emails to keep

  • keep emails with related digital information in a shared corporate information management system

  • limit what users can keep in personal email accounts by the use of: – email account quotas OR – automatic deletion after a set period of time

The National Archives of Canada

The following extract is taken from ‘Email Management Guidelines Roadmap‘ from The National Archives of Canada website.  The navigation path is Home/ Services and Program/Managing Government of Canada Information ResourceGuidelines/ Email Management

3.1. Maintain and use an organized and efficient filing system for email

Email messages, other than non-records or transitory messages, should be moved from the email system to a separate filing system where they should be organized as specified in the classification structure approved by the institution. Messages should be indexed and kept for institution use until their scheduled disposal or until their transfer to archival storage. Archival storage should also be organized and indexed for efficient retrieval.


Canadian courts have followed a well-recognized approach which holds that a document filing system that belongs to a party involved in litigation should be organized and labelled or indexed in such a manner as to facilitate use by the other party.

It can be extremely time-consuming to locate relevant evidence, and a party to litigation might argue that the cost of producing records for discovery is too great. Courts will weigh the cost of discovery against the potential benefits of having the evidence. However, Courts have not been sympathetic when it has been determined that a major portion of the cost is due to the fact that the party’s filing system is poorly organized.

It is important not to underestimate the cost of discovery. On a major case, involving a large institution, the cost can run into the hundreds of thousands, indeed into the millions of dollars. In a number of jurisdictions, some defendants have settled out of court, not as an admission of guilt, but because settlement was less expensive than assembling the evidence required for a defence.

National Archives and Records Administration of the United States (NARA)

The following extract is taken from NARA Bulletin 2014-06 Guidance on Managing Email

The navigation path is Home > Records Management > Bulletins >Bulletin 2014-06

 4. What is the role of Federal employees in email management?

Currently, in many agencies, employees manage their own email accounts and apply their own understanding of Federal records management. This means that all employees are required to review each message, identify its value, and either delete it or move it to a recordkeeping system. Some email, such as spam or all-staff announcements, may be deleted immediately. On the other hand, substantive policy discussions conducted in email may be appropriate for preservation for several years or ultimate transfer to NARA.

NARA recognizes that placing the responsibility on employees to make decisions on an email-by-email basis can create a tremendous burden. As a result, NARA recommends that agencies immediately begin to adopt automated or rules-based records management policies for email management, such as the Capstone approach.

5. What is Capstone?

Capstone is an approach to managing email. It is not a type of technology. (See NARA Bulletin 2013-02: Guidance on a New Approach to Managing Email Records.) When adopting the Capstone approach, agencies must identify those email accounts most likely to contain records that should be preserved as permanent. Agencies will determine Capstone accounts based on their business needs. They should identify the accounts of individuals who, by virtue of their work, office, or position, are likely to create or receive permanently valuable Federal records. Capstone officials will generally be the top-level senior officials of an agency, but may also be other key decision makers at lower levels of the agency.

Following this approach, an agency can schedule all of the email in Capstone accounts as permanent records. The agency could then schedule the remaining (non-Capstone)email as temporary and retain all of them for a set period of time based on the agency’s needs. The Capstone Bulletin addresses additional options and best practices.

The management of e-mail in UK Government – the comments of a civil servant

Getting off the train tonight I bumped into a civil servant acquaintance of mine. He has been working at a reasonably senior level for the past two decades and has been working on a reasonably important policy initiative for the past three years

He asked me what I was doing these days.

I told him I was still doing my records management consulting but I had also started a part time PhD

He asked me what I was doing my PhD research into.

I told him I was researching how the UK government was managing e-mail.

He started to laugh, Not a giggle, a proper belly laugh.

I asked why he was laughing

He told me that he didn’t think there were any proper government records since the break up of the paper record systems in the late 1990s. His Department used to use a massive electronic records management system, but ‘no-one could find anything in it and hardly anyone used it’.

The Department had ceased using that electronic records management system a year ago. They had intended to implement a new records systems based on SharePoint but it had been delayed so they have been making do with e-mail accounts and shared drives.

He said that the last time he had saved something into a central records system of any kind was five years ago.

All his correspondence was in e-mail. He has 11,000 e-mails in his account. His department used to routinely delete e-mails that were over 2 and a half years old but that practice was in abeyance since they abandoned their electronic records system.

I asked what he thought he would be able to hand over to his successor in the event that he left the department or transferred post. He answered ‘almost nothing’. He said that the final outputs of his work were available on his Department’s website, but the thought process of how he got there will be lost.

Then he corrected himself and said ‘actually it won’t be lost, it will exist on a server somewhere, it will just be completely unfindable’

Most UK government departments seem to delete the contents of an e-mail account some months or years after a civil servant leaves so I suspect it won’t sit on a server for terribly long if/when my acquaintance does leave the department.

The part time PhD I am embarking on is jointly supervised by the University of Loughborough and The National Archives.

Why were corporate wide records systems in the paper age so much more successful than those in the digital age?

Records managers are often accused of trying to replicate ‘a paper paradigm’ in the digital world.  This is a little ironic. If we were able to implement corporate wide electronic records management systems that were half as good as the best records systems in the paper days then we would be very popular indeed.

Our predecessors in the records management profession 20 years ago  tried to abstract the qualities of the best paper records management systems and express these qualities as a set of technology neutral criteria, in the hope and expectation that we would be able to design electronic records mangaement systems that also met those criteria, even if they met them in a completely different way.

The best statement of these technologically neutral criteria can be found in  section 8.2.2. the ISO 15489 records management standard, (see my last post for a more detailed discussion of them)

The five characteristics are as follows. In order to be considered reliable a records system must:

  •  routinely and comprehensively capture all records arising from the activities that it covers
  • act as the main source of reference for the activities it covers
  • link records to the activities from which they arose
  • protect records from amendment or deletion
  • preserve access to records over time

These characteristics may at first sight seem utopian. No organisation I know of currently operates a corporate wide system that meets all these characteristics and covers all of their activities. And yet in at the time they were drawn up, in the early 1990s,  they seemed anything but idealistic. Before the introduction of e-mail, any organisation that wished to could set up a record systems that met all five of these characteristics.

What made it possible to set up a reliable corporate wide records system in the paper age?

In the paper age there was a gap in time and space between:

  • the point in time at which a business document/communication arrived in the organisation from outside AND
  • the point in time at which that business document/communication arrived in the in-tray of the individual responsible for dealing with that communication

There was also a gap in time and space between

  •  the point at which an individual within the organisation sent a business document/communication out AND
  • the point at which that communication either arrived at the colleague it was addressed to, or left the organisation for dispatch to an external recipient

Organisations could insert control points into those gaps in time and space to ensure that business communications were routinely captured into the records system, and assigned to containers (usually called ‘files’) that each represented an instance of a particular activity.


How a registry system typically worked


The illustration above shows how such a registry system would typically work

Incoming post would arrive in a post room. The post room staff would do a rough filter of the post

  •  Things that looked like they were not needed for the record system because they were trivial, personal, or reference material (promotional material/flyers/postcards/love letters/magazine subscriptions) would be sent direct to the individual concerned
  • Things that looked like business communications (letters, memoranda, reports etc.) would not be sent direct to the addressee, instead they would be sent to the relevant records registry

Each registry was simply a team of records clerks who looked after the files for the area of the organisation within the scope of their registry. Typically an organisation would have several registries, each covering one or more of the organisationsfunctions/departments/buildings, though it was also possible to operate with one central registry covering all activities.

The registry would assign the document to the file representing the activity from which the correspondence arose. They would deliver the file, with the new correspondence on it, to the action officer.

The action officer would draft a reply which would be typed up by a typist in a typing pool. The typist would create two copies for the action officer to sign – one to go on the file, and one to be sent out.


Evaluating registry systems against the five criteria for a reliable records system

The registry system described above meets the five reliability criteria for a records system because:

  • There is routine and comprehensive capture into the record system.   In the post room(s) the same staff filter incoming correspondence day after day. They apply the same thought process to post day after day to decide what post goes to which registry, and what post bypasses the registries and gets sent direct to individuals. If the post room acquires a new member of staff, they train that person in the thought process.     Similarly in the registry(ies) the same staff do the filing day after day.   The registry staff have no interest in withholding embarrassing material from the file. The file is not holding them to account, it is holding action officers to account. The files are comprehensive – every incoming  piece of correspondence goes through the post room, and is either filtered out or sent to a registry for filing.
  • The colleagues working on the project/case/relationship/matter use the ‘file’ as their source of reference. If there are gaps in the file a colleague is not only likely to notice, there is also a fair chance that they will be motivated to do something about it, because they rely on that file being complete in order to be able to do their work, and defend their work.
  • A file is set up every time a new piece of work starts and every piece of correspondence placed on the file is, by the act of being placed on a file, connected to the activity it arose from.
  • The registry guarded the files. They typically kept a record of who each file was loaned out to and when. It is true that there was nothing to physically prevent an action officer ordering up a file, removing a paper that was incriminating to them from the file, and returning the file. However they would risk dismissal if detected.
  • Organisations had ways of managing the records lifecycle so that access to records were preserved over time. The registries would store active files close to office space, then send records to a record centre when they became non-active at a point some time after the work had finished From the records centre records would be disposed of at the end of a designated retention schedule either through destruction or through transfer to an historical archive

What happened to these systems when e-mail and networked computers arrived?

On the face of it it seems that the records management and archives profession was in a good position circa 1993 to ensure that no paper registry system was decommissioned without an adequate digital replacement being put in-place. Those organisations that operated such systems tended to proud of the systems, and proud of the records that the systems held. Their records were their support, their defence and their source of reference. They had no plans to jeopardise the quality of their records.

So why did that not transfer?  Why is it that even organsiations that had great records systems could not replicate the quality of those record systems after e-mail?

There are three main reasons to this:

  •  the speed and manner in which an e-mail moves through space and time is so different from that of a piece of paper that even having abstracted the qualities of a good paper records system into a set of criteria it was hard for the profession to imagine a way in which a system in the post e-mail world would meet those criteria
  • individual e-mail accounts collapsed the  time and space between a piece of correspondence arriving in an organisation and it arriving at the desktop of the action officer.    There was no time or space for records management controls to be inserted
  • the rapid and uniform spread of e-mail, through standard e-mail client and server software, meant that organisations denied themselves the opportunity to innovate when they set up their systems for handling e-mail.   A satisfactory method for transparently filtering and classifying/ filing e-mail never emerged because so little experimentation was done.


Evaluating automated approaches against records management principles

As a profession we are very proud of our principles.  If you ever discuss technology with a group of records managers one of us is bound to say  ‘records management principles are timeless, regardless of how much or how quickly technology changes’.

But what exactly are these principles?

The time when you most need a set of principles is when new ideas, tools, technologies or approaches come onto being.   By definition we have little or no practical experience of these new ideas/tools/ technologies.  We need a set of principles which distills our past experience of what has and hasn’t worked, in order to predict whether these new approaches are likely to work.

NARA’s report on automated approaches to records management

In March 2014 the  US National Archives (NARA) issued a report on the different ways in which records management could be automated.   The approaches have little in common with each other except that they all aim reduce the burden of records management tasks on end users as compared with the more established electronic records management system approach.

The approaches discussed in the report were extremely varied, but can be boiled down to the following:

  • an in -place approach based on holding a records classification and related retention rules in one application, and applying them to content held in the various different native applications of the organisation (SharePoint, e-mail accounts, shared drives etc.)
  • a workflow approach –  the definition of workflows for each activity, that include provision for the capture of records at particular points of the process
  • a defensible disposition approach – the definition of policies to apply to aggregations of content around the organisation.    For e-mail accounts you might have a policy that the e-mail accounts of individuals deemed to have important roles in the organisation are kept permanently, whilst the e-mail account’s of less important individuals are kept for six years.  You might set a policy that content in SharePoint sites/shared drives is deleted after five years/fifteen years. or kept permanently depending on the importance of the functions carried out by the team concerned.   There is no retention schedule, and no records classification, just risk based decisions.
  • automated filing by a rules engine  through the definition of rules to enable a rules engine to recognise which content arises from which particular activity
  • automated filing by an auto-classification tool  the  use of machine learning to learn the attributes of content that arises from particular activites,  without the organisation having to write a rule set.

(I have taken the liberty of changing NARA’s categories slightly – this summary  sticks more closely to NARA’s definitions)

 These new approaches are very different from the precious electronic records management system approach, but there has been no change in records management theory in between the two approaches (unless you count the new ideas from the information governance world – such as ‘everything held is a record’  ‘big buckets are better than granular hierarchies’ ‘end-users should not have to bear the burden of records management’)

NARA gives the pros and cons of each automated approach, without favouring any one or other of them, and without stating whether or not they believe each approach will keep records to an acceptable standard on a corporate scale.    This is not NARA’s fault – it is simply a recognition of the fact that at the moment we as a profession have no handy set of criteria to evaluate these approaches against.

In this blogpost I am going to nominate what I think is the  most useful and concise set of criteria for judging a records management system or approach- namely the five characteristics of a reliable records system that were developed in Australia in the early 1990s and ended up as section  8.2.2 of  the International Records Management Standard ISO15489.

The five characteristics of a reliable records system

Reliability is the most important characteristic of a record system.  A record system exists to perform a paradoxical function.  It exists to both:

  •  enable external stakeholders to scrutinise an organisation AND
  •  to enable the organisation to defend itself against external scrutiny

This paradox is fractal – it is present at any level of aggregation:

  • a record system enables an organisation to scrutinise a team, and it enables the team to defend itself from scrutiny
  • it enables an individual to be scrutinised and to defend him/herself from scrutiny.

The only way that a records system can resolve this paradox is by being trusted by all parties – in other words for all parties to consider the system to be reliable  – the individuals carrying out a  piece of work, their immediate colleagues and line management, the rest of the organisation, and external stakeholders.

Section 8.2.2.of the ISO 15489 standard states that in order to be considered reliable a records system must:

  • routinely and comprehensively capture all records arising from the activities that it covers
  • act as the main source of reference for the activities it covers
  • link records to the activities from which they arose
  • protect records from amendment or deletion
  • preserve access to records over time


Why the five characteristics of reliability are essential, rather than merely desirable

These characteristics are not ‘nice to have’ they are ‘must have’.

Think what would happen if a records management system did not meet even one of these characteristics:

  • If records were not consistently captured into the record system then external stakeholders could wonder wether there was a bias in record capture, for example whether content that could be incriminating/embarrassing was deliberately kept off the record system
  • If a record system does not comprehensively capture all records arising from an activity then this will leave gaps in the record and weaken the organisation’s ability to defend or learn from the way it carried out that activity.  It will also lead to external stakeholders looking to other sources of information outside of the records system – sources that may be more time consuming and more embarrassing for the organisation to search.
  • if a record system does not serve as the main source of reference for the activities within its scope then any gaps in the record will go unnoticed, and uncorrected.   It is particularly important that the record system is used as source of reference by colleagues carrying out the piece of work itself – because they are the only people in a position to know that there is content missing from the record.
  • if records are not linked to the activity from which they arose then the organisation will find it impossible to apply a precise retention period to those records.   Retention periods are specific to a type of activity:  managing staff,  designing a building, managing a project, bringing a pharmaceutical product to market,  adjudicating on a citizen benefit claim etc..  The trigger point for that retention period to start is even more specific, it is specific to a particular instance of each activity:   the date a particular member of staff left employment;  the date a particular building ceased to exist; the date a particular project finished, the date a particular pharmaceutical product was withdrawn from the market, the date a particular citizen ceased receiving benefit etc.
  • If records are not protected from amendment or deletion then an external stakeholder will have cause to doubt whether or not content detrimental to the organisation, or to a particular team or individual, has been deleted from the system prior to them viewing the record
  • If access to records is not preserved over time then an organisation cannot be sure that if will be able to defend itself from scrutiny/challenge if that scrutiny or challenge comes at a date in the future.  By the same token external stakeholders cannot be sure that they will be able to scrutinise the organisation at a future date.


Why have we lost site of these reliabilty criteria?

At first site it seems odd that we as a profession have lost site of these criteria for judging a records system.

These criteria are not obscure.    They were a fundamental part of the drive by the records management and archives professions to manage the transition from the paper age to the networked digital age by expressing the attributes of good recordkeeping systems in an abstract, non-format specific way.

Although these criteria are trying to be as timeless as possible, they are also very much of their time.  They were written at the start of the 1990s, just before the mass adoption of e-mail and networked computers with shared drives  by organisations in what were then called developed economies. They were embedded in the Australian records management standard which later became the International Records Standard (ISO 15489, published in 2001).

The reason why we have lost site of them is that we have not been able to implement systems that meet all five of these criteria,  on the scale we want to work at (the corporate scale) since the mass adoption of e-mail.  Nor is there a realistic likelihood that any of the five automated approaches discussed by NARA will meet all five of these criteria.


Evaluating existing record keeping systems against the five reliability criteria

When you compare existing systems within organisations to these reliability criteria you find that:

  • line of business systems (case file systems and sector specific systems such as insurance claim systems) can be set up to routinely meet all five reliability criteria.   They can use functional e-mail addresses, and/or web forms, to divert correspondence related to that function away from private e-mail accounts directly into the system, which means they can routinely and comprehensively capture correspondence arising from  the activities that they cover.   However each line of business system can only cover one area of the business.    Organisations carry out  so many different types of work that it is impossible to have a line of business system for each of them.
  • generic document management repositories such as shared drives, electronic records management systems, and SharePoint cannot routinely and comprehensively capture business correspondence sent and received through e-mail – they are dependent on individuals exercising their own judgement on which e-mails (if any) they upload or push to the system.    None of these three types of repository serve as the main source of reference to all or most of the activities that they cover.    Individuals tend to use their own e-mail accounts as the first place  to go to find records of their work.   Of those three repositories electronic records management systems and, to a lesser extent SharePoint, do a better job than  shared drives at protecting records and linking them to the business activities that they arose from.
  • e-mail archives routinely and comprehensively capture electronic correspondence.  However they do not relate records to a business activity which leads to problems with applying access and retention rules .  They cannot be used as the main source of reference for activities by anyone except Legal Counsel because private and sensitive e-mails are undifferentiated from other correspondence


Evaluating automated approaches against the five reliability criteria

The automated approaches described in the NARA report  either don’t meet all five reliability criteria or they don’t scale across an organisation

  • the in -place approach – currently lacks an answer to the question of how  to routinely and comprehensively capture important e-mails and relate them to business activities.  It is dependent on either individuals dragging e-mails into folders (which is not routine or comprehensive because many individuals never file e-mail into folders) or one of the automated filing approaches (see below)
  • a workflow approach – works well, meets all five criteria, but cannot scale across an organisation because of the time taken to analyse processes and define workflows
  • a defensible disposition approach – does not protect records.   A disposition rule is applied to a part of a shared drive or a SharePoint site but indviduals can delete or amend content before the time at which the disposition rule is applied.
  • automated filing by a rules engine  – the time spent to write the rules means this approach does not scale to a whole organisation
  • automated filing by an auto-classification tool –  a typical organisation carries out a great many different types of activity.  For each type of activity it carries out a great many different instances of that activity.  An auto-classification tool  has to be trained in each container/category it is asked to file records into.   The more containers/categories you have the higher the cost of training.  Organisations trying to implement auto-classification corporate wide have had to compromise and ask the tool to file into ‘big buckets’ rather than into a container for each instance of each activity (project/case/matter/ relationship etc.).  This means that on a corporate scale auto-classification does not currently meet the criteria of linking records to the business activities that they arose from.  I explained above that although retention rules may be set on broad swathes of activites (e.g records of all our projects are kept for ten years after the project ends) to apply that rule you need to have each record allocated to a particular project so that the ten years can be triggered by the end date of that particular project.   The buckets also end up being too big to be navigable or useable by end users, meaning that the system ends up not being used as the main source of reference for the activities it relates to.



At this point in time we need to be honest and say that we have no approach to implementing systems on a corporate scale that will routinely and comprehensively capture business correspondence , protect it, link it to the business activity it arose from , maintain access to it over time, and  serve as the main source of reference for the activities it covers.  Nor have we the prospect of such an approach any time soon.

Until we get such an approach organisations will suffer problems with their records management/information governance.

We have seen organisations establish electronic records management systems/SharePoint implementations and hope that access to information requests/ e-discovery can be confined to those systems, only to find themselves searching e-mail accounts, shared drives and maybe even back up tapes.  This is because their official records system does not routinely and comprehensively capturing records.  Their external stakeholders have responed in effect by treating all the organisation’s applications as being part of their records system.

We need approaches to records management that result in systems that are both reliable and relied upon.   If end users do not rely upon a records system,  but instead refer mainly to content outside the scope of the record system, then they will neglect to point out omissions in the record system, and there will be a disconnect between the records available to the individuals carrying out the work, and the records available to those wishing to scrutinise or continue their work.