When I started telling people, back in 2015, that I had started a doctoral research project on email, my records management colleagues would typically smile, pat me on the shoulder and say ‘great! let us know when you have found the solution’.
Two years later. There is still no solution. Solutions do not drop out of the sky. Solutions emerge from people trying to fix things. If there is a broken situation and noone tries to fix it, it stays broken.
You do not have to undertake a doctoral research project to work out that organisations are not consistently capturing business email into systems that they designate as ‘records systems’.
Yes there are potential approaches. Not just one, many different approaches, suitable in different circumstances, using different technologies or different ways of harnessing human input:
- you could do what some CRM systems do and match the email addresses of external contacts with locations in your record system (when colleague a sends an email to person/organisation b it is about business matter x, when they email person/organisation c it is about business matter y etc.);
- you could set up one container in your record system to mirror each email account that you provision. You could give each email account owner a simple means of indicating for every email they send, whether or not they want that email and the thread it is part of to be saved as a record to the mirror container for their account in the record system;
- you could use analytics just before you run your routine deletion routine on an email account (after someone has left, or after two or three years) to rescue material which is obviously business related (and obviously not personal) from routine deletion.
We could go on and on with this list, you will have ideas of your own. We have not even mentioned machine learning yet.
But a solution for a problem is like a hammer waiting for a nail unless the problem owner wants the problem to be solved.
The problem owner for the email ‘problem’ is the organisation who owns the email accounts and the emails.
The one thing all of the above solutions have in common is that they all result in more emails being captured into record systems. All of these solutions therefore add to storage costs, they also add to the cost of servicing access to information requests, and, in highly politicised environments such as many central government departments, they add to the perceived risk of being forced to disclose potentially embarrassing content.
Email is therefore a problem that the problem owner percieves a benefit from leaving unsolved. Every conceivable solution to the problem of treating business emails as records would be more expensive, and would be perceived by the problem owner as being more risky, than the current approach. The current approach, if we are honest, is for organisations to make a token effort to persuade staff to save important emails as reords, who in turn make a token effort to occasionally place a few emails into a record system, whilst all the rest of the emails are routinely deleted after a designated but essentially arbitrary time period.
Where does this leave archivists and records managers? Where does it leave records management and where does it leave record keeping?
We are also in a manner of speaking the problem owner. The problem falls into our professional domain.
We are employed by organisations. It is not our job to try to pursuade our organisations to do anything that is against their perceived interests (and we would not get very far if we did try).
On one hand we are in a good situation. Organisations put us under no pressure to change our records management policy telling staff to move emails into the record system, and under no pressure to change the relationship between the email environment and our record system (which is typically a product primarily designed and configured for the capture and management of documents rather than emails).
On the other hand it puts us in a very difficult position. We go to all that trouble to develop corporate wide record systems and corporate wide records retention schedules (both labours of Hercules, demanding all of our professional skills, experience and energy) and yet we are regarded as failures because those systems and those retention schedules fail to embrace the bulk of an organisation’s business correspondence.
It is a problem that in effect we can do little about, because our organisations do not want to do anything about it. Take any of the ideas listed above to your organisation and you will see what I mean. Our vendor community cannot help us because our organisations do not want solutions that would result in them keeping significant volumes of email for significantly longer periods of time.
In the short term we can carry on like this. Our organisations are happy for us to continue to try and continue to fail. Because (and this is the paradox) our failure to capture business correspondence consistently into systems treated as record systems succeeds in reducing the cost and perceived risk of recordkeeping to our organisations.
But what about the long term? I fear in the long term that this success-through-failure damages our professional reputation and damages the clarity and moral force of our theory and of our practice.
The story that the Financial Times  and other newspapers ran last week about the Cabinet Office 90 day email deletion practice is not new. It was reported on 18 December 2004 by the BBC, two days before the deletions were about to start. Here is the first part of their report 
I tweeted a link to the BBC report on Saturday . I received this tweet in reply
Imagine yourself back on the 18 December 2004, a fortnight before the UK’s Freedom of Information Act comes into force. It comes to light that the Cabinet Office, at the very heart of Government, is planning to have all emails that are over 90 days old deleted from their email servers. In other words they are deleting about ten years worth of correspondence – a portion of which will have been captured elsewhere, but much of which will only have existed on those email servers. The opposition leader has raised objections. Why did archivists and records managers not raise this as an issue?
There are four main reasons why archivists/records managers did not object at the time:
- We had spent the previous ten years warning our organisations about the inconveniences and dangers of records building up in individual email accounts. We therefore were not minded to defend those records that had built up in Cabinet Office email accounts
- We did not know what to do with email accounts. We did not know how we would manage them over time, how we would deal with the personal data within them, how we would sensitivity review them, what retention rules we could apply to them, nor how we would appraise them as being worthy or unworthy of permanent preservation. We could not envisage how or when we could make available to the public an email account selected for permanent preservation.
- Records management in the UK had a great year in 2004, mainly thanks to the Blair government. During 2004 central government threw money at electronic records management systems, created lots of new records management posts, employed lots of records management consultants. It was exciting. We were not in a mood to question things too closely.
- On December 18 2004 we were all so focused on the coming of FOI (and perhaps the coming of the Christmas holiday) that we didn’t notice. I don’t remember even seeing this article, I can’t remember anyone mentioning it to me.
This is an important case with significant implications for records management and archival practice. We might usefully debate the following questions:
- Was the mass deletion of Cabinet Office emails from email servers in December 2004 carried out for political advantage, administrative expediency or for recordkeeping improvement? Has the Cabinet Office continued that 90 day auto-deletion policy under successive Labour, Coalition and Conservative administrations for political, administrative or recordkeeping reasons?
- How serious is the impact of these deletions on the historical record? Is the important correspondence largely captured elsewhere? Or is this 90 day auto-deletion of email by one of the most important Whitehall Departments going to create a significant and irreplaceable gap in our nation’s historical record?
- Are we as a profession – records managers and archivists – any better equipped to manage email accounts over time now than we were back in 2004? Is there a feasible alternative to auto-deletion?
- Should the UK National Archives follow the example of its counterpart in the US and step in to prevent the auto-deletion of significant email accounts by declaring that it requires UK Government departments to select the email accounts of important civil servants for permanent preservation?
 Pickard, Jim and Stacey, Kiran 16 June , 2015 8:17 pm Freedom of information is Mission Impossible for Downing St emails. Financial Times, , available from http://on.ft.com/1QCbS8a (it requires a log-in). Accessed 17 June 2015 OR see see Morris, Nigel 17 June 2015. Government faces call to review self destruct email policy http://www.independent.co.uk/news/uk/politics/government-faces-call-to-review-selfdestruct-email-policy-10327202.html (accessed 20 June 2015).
 BBC News, 18 December 2004, 14:56 GMT. Howard condemns email deletion. Available at http://news.bbc.co.uk/1/hi/uk_politics/4107563.stm (accessed 20 June 2015).
The Financial Times  reported this week that the Cabinet Office auto-deletes their e-mails after 90 days.
In the article several officials and aides are quoted describing day to day problems that Cabinet Office staff have experienced in the absence of an adequate correspondence record:
- A special advisor is quoted as saying that ‘It means that people don’t remember things…It is dysfunctional. Then they check their emails and they don’t exist anymore’.
- An aide is quoting as expressing their frustration that colleagues often had different recollections of what had been agreed at meetings.
The Cabinet Office’s 90 day deletion policy was introduced in 2004. The Freedom of Information Act for England and Wales came into force on the 1 January 2005. The article quotes Maurice Frankel of the Campaign for Freedom of Information as stating that the proximity of those two dates ‘was not a co-incidence’. However the article also reports:
An official said the system had been recommended by the National Archives for ‘best records management’ and it was a coincidence that it began at the same time as Freedom of Information
On the one hand I do not believe that the Cabinet Office’s 90 day deletion policy was introduced in 2004 purely as a conspiracy:
- In 2004 many Whitehall departments were introducing electronic records management (ERM) systems, into which they told staff to capture significant documents and correspondence.
- It is quite possible that the Cabinet Office’s routine deletion policy dates from the time of their roll out of their ERM system
- The UK National Archives advice was at the time (and still is now)  that Departments should capture significant correspondence into a designated records system and should routinely delete correspondence from e-mail accounts.
- As FOI man  points out, back in 2004 most records managers would have recommended that an organisation implementing an ERM system should also routinely delete e-mail left in email accounts
- At the time, in 2004, we as archivists and records managers genuinely hoped and believed that the policy of requiring colleagues to declare important emails and documents into an ERM system would capture a reasonable record of their correspondence
- The ERM systems introduced around 2004 by many UK government departments, including the Cabinet Office, had been tested and certified by the UK National Archives as meeting their specification of requirements for the management of electronic records
- The UK government was the first administration in the English speaking world to implement ERM systems on a large scale. If felt like an experiment at the time – powered by the Blair government’s modernising government agenda, the coming of FOI in England and Wales (and separately in Scotland), and by the fact that paper records systems had been blown apart by the coming of email in the mid 1990s.
On the other hand :
- just because archivists and records managers in the UK government sector and elsewhere recommended the routine deletion of email from email accounts in good faith, back in 2004, whilst embarking out on the electronic records management system journey , and in the hope and on the assumption that the policy instruction to save significant emails into electronic records management systems would be followed….
- …..does not mean that we should continue to recommend or support such routine deletions now, in 2015, in the light of the experience that we have gained in the intervening decade.
Experience has taught us that asking users to declare important emails as records into any sort of record system does not result in the capture of an adequate correspondence record
The experience we have had in the UK, in continental Europe, in Australia and in the US is that asking colleagues to choose which of their e-mails are significant, declare them as a record, and move them to some sort of electronic file (or paper file, or SharePoint document library) does not capture an even vaguely adequate record of correspondence.
It is interesting to read the quotes of the special advisors/aides/officials in the FT article. One presumes that some emails have been captured into the Cabinet Office’s ERM system (for nearly a decade they had a system provided by Meridio, but have recently replaced it with Google Docs). And yet the officials quoted by the FT talk as though there was no correspondence record at all. My guess is that this is because individuals saved so little correspondence into the system, and saved into the system so selectively, that to all intents and purposes there was and is no Cabinet correspondence record outside of e-mail accounts.
This is only a guess – we would need to know the figures. The most relevant figures being:
- how many e-mails did Cabinet Office staff send in any given year?
- how many e-mails were captured into the Cabinet Office record system in that same year?
We do have these figures for the US State Department, thanks to a report  published by the Office of Inspector General in March 2015. The report states that in 2009 the State Department implemented a record system that was built into the Outlook email client, whereby individuals could declare important emails as records. This was backed up by a policy instruction that individuals were responsible for preserving important emails as records, and by training.
The Office of Inspector General’s report sates that in 2011 the State Department sent 1 billion emails, but only 61,556 were declared as records. Robert Smallwood  calculates that the emails captured into the State Department’s record system constitute only .0061% of their total email communications. In 2013 the situation was even worse – only 41,749 State Department emails were declared as records. Interestingly the report states that one of the reasons for the lack of the declaration of emails as records was a fear that they would show up in searches (which I am interpreting to mean that they would show up in searches conducted in response to Freedom of Information requests).
If the Cabinet Office no longer knows how many emails it received in a given year then the headcount of the Cabinet Office in that year would be the next best figure. Chris Prom  quotes statistics from Radicati  stating that on average a business user sends 33 emails a day.
I recently spoke to an organisation who had a reasonably good electronic records management system, no fear of Freedom of Information, and a strong commitment to record keeping. They told me that the number of items contributed to their records system equated to one item per member of staff every two days. If the figures Prom quotes are correct that would give a ration of around 1 email saved to their records system for every emails 66 sent. That is several orders of magnitutude better than the State Department’s ratio, but for an important organisation that ratio is still nowhere near high enough to function as a useable and defensible correspondence record.
Most UK government bodies operate routine deletion policies, albeit significantly less drastic than the Cabinet Office’s 90 day deletion. Rather than a 90 day deletion it is more common for the deletion to occur 1 or 2 years after the email is sent, or six months to a year after the individual email account holder has left employment. Alternatively or additionally some departments operate a limit on email box size rather than an automatic deletion.
Individuals in such departments do not experience the inconveniences reported by Cabinet Office staff. They are cushioned from the impact of the deletions, but the net result is the same. The correspondence record being kept is inadequate and cannot support basic succession planning/staff handover in the short term, corporate memory in the medium term and historical accountability in the long term.
 Pickard, Jim and Stacey, Kiran 2015. Freedom of information is Mission Impossible for Downing St emails. Financial Times, June 16, 2015 8:17 pm, available from http://on.ft.com/1QCbS8a (it requires a log-in). Accessed 17 June 2015
 The National Archives (TNA) (date unknown) Managing emails. Available from http://www.nationalarchives.gov.uk/information-management/manage-information/policy-process/managing-email/ (Accessed 18 June 2015). TNA’s policy advice to Government departments is that
”You will need to…..limit what users can keep in personal email accounts by the use of:
– email account quotas OR
– automatic deletion after a set period of time”
 Gibbons Paul (FOI man) 17 June 2015. FOI avoidance or good records management? Cabinet Office Email Policy. Available from http://www.foiman.com/archives/1584 accessed 18 June 2015
 Office of Inspector General, March 2015. Review of State Messaging and Archive Retrieval Toolset and Record Email (ISP-I-15-15) Available from https://oig.state.gov/reports/inspection?page=1 or access the pdf directly from https://oig.state.gov/system/files/isp-i-15-15.pdf
 Smallwood, R, 2015. Clinton Email Kerfuffle: Call for Cabinet-level Information Governance. Available at https://www.linkedin.com/pulse/clinton-email-kerfuffle-call-cabinet-level-governance-smallwood(accessed 17 June 2015)
 Prom, Christopher J, 2011. Preserving email (Digital Preservation Coalition Technology Watch Report) pdf available from http://www.dpconline.org/newsroom/latest-news/805-email-tomorrow-and-next-year-and-forever-preserving-email-report-published On page three of the report Prom writes ‘Over 3.1 billion email accounts currently exist, and the average business user sends 33 email messages per day (Radicati Group, Inc. 2011a). ‘
 Radicati Group, Inc., 2011a. Email Statistics Report, 2011–2015, Executive Summary. Available at: http://www.radicati.com/?p=7261 [Accessed July 17, 2011]. (Quoted by Prom)
I attended the Information Governance and eDiscovery summit in London last week.
Two friends, both stalwarts of the records management world in the UK, separately came up to me and said ‘you know what I think Records management is dead now James’
Records management isn’t dead. It is a contestable space at the heart of information governance.
In the e-Discovery stream of the summit there was no mention of any type of document management or records system (whether that be an electronic records management system, or SharePoint). This is because these systems are often not relevant to the preparation of lawsuits, where the lawyers go straight to the e-mail record and/or text messages to establish the chain of events and discussions pertinent to the case.
In effect organisations are operating two parallel spheres of recordkeeping.
- the communications/correspondence sphere that consists mainly of their e-mail servers/e-mail archive, but also any capture of instant messaging and/or text messages
- the document sphere that consists of whatever combination of electronic records management (ERM) system/SharePoint/line of business system/fileshare applications they deploy
The key difference between the correspondence/communication sphere and the document sphere is that:
- an e-mail server/archive can be set up to capture all the traffic going to and from a particular e-mail account on a particular server. Similar arrangements can be set up for text messages on company devices and for instant messaging systems.
- an organisation wide ERM system/SharePoint implementation captures only those documents/communications that an individual chooses to declare to that application.
The paradoxical relationship between the two spheres of recordkeeping
This split between e-mail servers/archives on the one hand and corporate ERM systems/SharePoint implementations on the other creates the following paradoxes:
- Organisations fear the contents of their e-mail archive/servers but lack confidence in the completeness of their designated record repository (ERM/SharePoint etc.)
- An implementation of an ERM system/SharePoint or similar has an advantage over an e-mail server/archive in that the information in it poses a lower risk to the organisation. Sensitive personal information is likely to be confined to predictable places within the system, and unguarded comments are less likely to find their way onto the system.
- an e-mail server/archive is more effective than an ERM system/SharePoint in enabling an organisation to account for itself in difficult or controversial circumstances. For example in situations such as litigation, investigation, or hostile freedom of information requests. However an e-mail server/archive is not useful for non- contentious, day to day internal usage, because the undifferentiated presence of sensitive personal communications means we cannot allow colleagues to access/search each other’s email accounts.
- An ERM system/SharePoint is good for non-contentious day to day internal usage but is weak in times of dispute, litigation and investigation.
- Some organisations configure their e-mail server/archive so that it retains a copy of every message even after an individual has deleted that message from their e-mail client. This best ensures they have have a complete record of e-mail correspondence and thus best supports accountability. However it also means that they are more likely to hold unguarded, trivial and sensitive personal e-mails because the individual account holders will not be able to delete them
- The weakness of corporate ERM systems/SharePoint implementations is that they are incomplete. They do not contain relevant documents or correspondence if the individuals who handled them lack the time, motivation and/or awareness to put them onto the system.
- Organisations are operating two separate sets of governance arrangements for these spheres Records managers might typically set the retention rules for the ERM/SharePoint sphere. IT managers and/or legal managers might set the rules for the retention of communications/correspondence stored in e-mail accounts and for instant messages and text messages.
- Most records management retention schedules are written as though they are universal and format neutral. They often make provision for relatively long retention periods for certain types of correspondence (ministerial correspondence, correspondence with regulators, correspondence with foreign powers etc.). And yet these retention schedules are rarely applied to the e-mail accounts which hold most of that correspondence, and which are often deleted after a relatively short time period. Note that NARA’s recently issued retention schedule for e-mail accounts of US Federal agencies is an exception to this rule (NARA, 2015)
The task of information governance
Information governance is often described as an umbrella term covering the separate disciplines of eDiscovery, records management, access to information and privacy.
The inclusion of both eDiscovery and records management under this umbrella means that information governance spans both:
- the communications/correspondence sphere (e-mail archives/servers and any other communications tools deployed) AND
- the document management sphere (ERM systems, SharePoint and other collaboration systems, shared drives)
One of two things could happen here.
One possibility is that each of the component disciplines of information governance stay entirely separate. Records management carries on asking people to move important correspondence to an ERM/SharePoint or whatever tool they designate as their main repository. Lawyers perfect their eDiscovery/analytics tools which allow them (and only them) to search across the email server/archive and across all repositories in the organisation.
The other possibility is that information governance acts to influence these disciplines, to create synergies and to enable them to become more than the sum of their parts. To an extent this is happening already as analytics tools honed on eDiscovery cases cross over into the records management space to enable organisations to apply disposition decisions to hitherto intractable repositories such as shared drives.
Information Governance should work towards becoming the arena in which the tensions and contradictions between accountability, risk, and privacy can be resolved or managed in relation to both the main correspondence system (which at the time of writing happens to be e-mail) and the main document management applications and repositories.
The task of records management
I would like to see records management use information governance to blur the boundaries between the systems they designate as records systems, and the other repositories in the organisation including, but not limited to, e-mail servers/archives. Lawyers use their eDiscovery tools to search across all repositories in the organisations. As records managers we should be seeking to stretch their retention remit to cover all repositories. We should be seeking to establish relationships between content held in whatever we designate as our main records repository and related content held on shared drives, in e-mail accounts etc.
Cutting through the e-mail paradoxes
Organisations would like to be in a position where they can dispense with the complete record of e-mail communications after as small a time interval as possible, and rely instead on the filtered, lower risk records that exists in their document management sphere. However the swift deletion of e-mails from their email server/archive adversely impacts their ability to account for their actions.
The irony is that external stakeholders/litigants/regulators/hostile FOI requesters are not interested in trivial e-mails, and they are not interested in personal e-mails. External stakeholders would be happy for organisations to delete or redact them.
The problem is that organisations cannot currently defend or explain how they got from the complete correspondence record on the e-mail server/archive, to the filtered record on a corporate electronic records management system/SharePoint/shared drive.
Records management should work towards providing organisations with a defensible consistent, routine and transparent way of distinguishing trivial, sensitive and personal e-mails from typical business correspondence, with a view to enabling organisations to have a record of each individual’s business correspondence that is both filtered and comprehensive; defensible and accessible; to which defensible retention rules can be applied, and which can be linked to related collections of documents and communications held in other applications/repositories, including the designated records repository (ERM/SharePoint or similar).
NARA (2015), GRS 6.1 Email Managed Under a Capstone Approach http://blogs.archives.gov/records-express/files/2015/04/FINAL-GRS-6.1-Review-Package-FR-Posting-03.30.15.pdf (accessed 19 May 2015)
Ravanbakhsh, Arian (2015) DRAFT Capstone GRS Available http://blogs.archives.gov/records-express/2015/04/02/draft-capstone-grs-available/ (accesssed 20 May 2015)