Why NARA has no option but to preserve significant e-mail accounts

390-capstone-pyramidOn 30 May 2013 the US National Archives and Records Administration (NARA) opened a consultation period on its proposed ‘Capstone’ approach to managing e-mail.  If the approach is adopted US federal agencies will be asked to schedule the e-mail accounts of senior staff for eventual transfer to NARA and permanent preservation.

‘Capstone’ is the term used for the apex stone on top of a pyramid.  The capstone is the only stone in a pyramid that looks out in all four directions.  On the US dollar bill the capstone of a pyramid is depicted as an all-seeing eye.

NARA are hoping that the e-mail accounts of senior managers act like the capstone of a pyramid.  They are working on the assumption that all major discussions and decisions in an organisation are filtered up through one or more members of senior management, and leave a trace in their e-mail accounts.

The nature of the Capstone approach to e-mail

Here is an extract from NARA’s description of their proposed Capstone approach:

Capstone offers agencies the option of using a more simplified and automated approach to
managing email, as opposed to using either:

* print and file .. or

* records management applications that require staff to file email records individually.

Using this approach, an agency can categorize and schedule email based on the work and/or position of the email account owner.

The Capstone approach allows for the capture of records that should be preserved as permanent from the accounts of officials at or near the top of an agency or an organizational subcomponent.

An agency may designate email accounts of additional employees as Capstone when they are in positions that create or receive presumptively permanent email records.

Following this approach, an agency can schedule all of the email in Capstone accounts as permanent records.

The agency could then schedule the remaining email accounts, which are not captured as Capstone, as temporary and preserve all of them for a set period of time based on the agency’s needs

What does Capstone mean for electronic records management approaches?

390-Capstone-MaryHeinrich-

The quote in the picture is from Marty’s blogpost on Capstone 

It is almost two decades since e-mail came into general office use in countries such as the US and the UK.   For most of the this period central/federal government bodies in major economies operated an ‘electronic records management system’ approach.   This involved implementing an electronic records management system (or rigging up a collaborative system such as SharePoint or Lotus Notes to act as one) and asking individuals to move documents or e-mails needed as records into that system.

Some such organisations tried to prevent staff ‘hoarding’ important e-mails in their accounts by either placing size limits on e-mail accounts, or automatically deleting e-mail from e-mail accounts after a defined time period (for example a year after receipt).

Such organisations would tell their staff that e-mail was a communications tool, not a recordkeeping tool, and that e-mail accounts were only a temporary storage space.  They would tell staff that anything needed in the medium and longer term must be moved onto a file in the records system.

NARA’s proposed Capstone approach effectively turns this on its head.  It asks US Federal agencies to treat e-mail accounts as records – records that will , for some key members of staff, be kept in perpetuity.

A senior member of staff who left all their e-mails in their e-mail account and never filed or deleted anything, would in theory, be meeting their accountability obligations to US citizens.

In practice Federal agencies are going to have to carry on asking staff to move (or copy) important e-mails out of their e-mail accounts, even after Capstone comes into implementation.  Capstone is not a records management solution. For example it does not solve the problem of staff being unable to access important e-mails sent/received by their predecessor.  This is because the accumulations of private communications in e-mail accounts make standard e-mail accounts unshareable.

Without e-mail accounts there will be a black hole in the records

390-StorageSpacePercentages -small

 Storage percentages given to me by a global multilateral institution 

To understand why NARA are proposing to permanently preserve e-mail accounts  let us look at what happens when an organisation implements an electronic records management system.

I recently heard a talk from a global multilateral institution who had a  strong records management programme in place. They had a good culture of record keeping stretching back over many years, longstanding senior management support, and a knowledgable and committed records and archives team.

They had been running an electronic records management system for over 15 years (first in Lotus Notes, then Documentum), which applied their corporate filing plan and retention rules to records.  They had got an integration between e-mail and the electronic records management system, and a route for colleagues to save documents needed as a record from SharePoint into the electronic records management system.

At one point in the talk they gave figures for the total amount of storage taken up by the different information systems in their organisation.   The electronic records system, with its retention controls, accounted for only 4% of the total storage.  A further 33% were taken up by other document storage applications (SharePoint, shared drives, line of business systems,  their website and intranet.)  An astonishing 63% of the storage was taken up by e-mail – 47% in e-mail accounts and 16% in their e-mail archive.

I spoke to the archivist of the institution and asked whether she was concerned that only 4% of the records of the institution were under the protection and control of their retention schedule and fileplan.  She said was indeed concerned, but she also pointed out in the days before e-mail, a national archive would typically only take between 3 and 5% of the records of government bodies- a similar figure to the percentage of documentation kept in that institution’s electronic records management system.

It was a good point, but there is an important difference.   In the paper days a well organised central government department/ federal agency would have retention schedules covering almost all of their records, all across the organisation.  The 3 or 5% of the records selected for permanent preservation in the relevant national archive was a distillation of the whole:  like a  capstone at the top of the pyramid of the organisation’s records.

In contrast the electronic records management approach leaves us with a situation where the vast majority of records (96% in the case of the institution referred to above) are outside of any retention control.

From an archivist’s point of view that would only be acceptable if the electronic records management system was routinely capturing the most important of the institution’s documentation.

But the (roughly) 5% of information/documentation captured by electronic records management systems  is not necessarily the most significant 5% of documentation for the organisation.

An electronic records management system will typically contain a records classification that covers all of the work of the institution/organisation.  Within that classification there will probably be a file (or a document library if their records system is built in SharePoint) for each significant piece of work undertaken by the organisation.

The problem is that these files will not be complete.  There will be swathes of correspondence arising from those pieces of work that never make their way onto the relevant file/document library.

This is because filing routines are not consistent – they vary with the motivation, workload and awareness of each individual member of staff.   This inconsistency leads to gaps in the files held in electronic records management systems.  The files are set up to tell ‘the whole story of a piece of work’, but they rarely do.

On a day to day basis individuals rely on their e-mail account rather than the relevant file in the electronic records management system, so they tend not to act to fill in gaps in the file (and may not even notice the gaps).

In launching the Capstone consultation NARA are, in effect, saying that they do not trust the electronic records management system/SharePoint implementations of Federal Agencies to act as the capstone on top of the pyramid of all documentation and correspondence.

For the world of e-mail they are asking for a separate capstone – the e-mail accounts of senior staff.

NARA has no real option but to ask US Federal agencies to preserve the e-mail accounts of senior figures. NARA has a duty to future generations to preserve the correspondence of people playing significant roles in federal agencies.  If such correspondence is not finding its way into the ‘official’ electronic records systems of those agencies then NARA needs those e-mail accounts.

Notes

  • For an insightful discussion on the Capstone proposal read Barclay T.Blair’s blogpost of June 21 2013
  • The latest update on Capstone I could find at the time of writing was this news piece from FCW published 21 August 2013
  • The FCW news piece states that NARA will say more about Capstone at this event NARA are running for the vendor community on the 10 September.   At the event NARA will issue ‘a grand challenge to industry’ regarding the type of technology needed with a view to ‘supporting Federal agencies as they implement the Managing Government Records Directive, particularly directive Goal A3.1.’ Goal A3.1 is the commitment ‘to work with private industry and other stakeholders to produce economically viable automated records management solutions’.    @adravan pointed out to me on Twitter that the event will be discussing not just Capstone but electronic recordkeeping challenges in general.

How long should an e-mail account be kept after a member of staff leaves?

On 30 May 2013 two postings appeared that between them shed light on how organisations are currently managing the archived e-mail accounts of staff who have left:

    • The first was a post by Rebecca Florence to the IRMS Records-Management-UK listserv that kicked off a debate on e-mail account retention and deletion
    • The second was a blogpost by Emma Harris of State Records New South Wales reporting the findings of a survey they had conducted into how public offices in NSW are managing their e-mail

Rebecca Florence posted a description of the situation in her organisation:

The current arrangement is that for a period of time post-leaving, access to the mailbox and email archive (in our case we use the Symantec Enterprise Vault) can be passed to a designated member of staff.

After that period of time has elapsed the mailbox/archive is deleted by IT, with the contents being exported to a separate restricted access area. Access is granted to the exported contents on a case by case basis. Currently the exported content is held indefinitely.

I should add that as you would imagine there are policies and guidance in place which advises staff to save emails where necessary outside Outlook for longer term retention and also assigning responsibility post-leaving allows for a review of any remaining emails for ongoing business use. I’m sure as most of you will have experienced, there is disparity across departments in regards to how well this is managed.

Phil Bradshaw replied that keeping records indefinitely is not the same as keeping records permanently:

  • keeping records permanently means we have assessed the records and found them to have enduring long term value
  • keeping records indefinitely means we cannot find a basis to set a retention rule on them

Is it possible to deal with e-mail by reviewing e-mail accounts when members of staff leave?

Lawrence Serewicz responded to Rebecca’s post by pointing out the legal costs and risks of maintaining all e-mail accounts indefinitely:

  • e-mail accounts generally contain personal data and the indefinite retention of entire e-mail accounts may  breach several of the EU data protection principles.
  • information held in an e-mail archive may be subject to discovery in the event of a legal case, and to disclosure in the event of an access to information request

Lawrence recommended that e-mail accounts get deleted three months after a member of staff leaves, but only after:

  • a pre-exit process in which the line manager and the employee go through the e-mail account together and decide how to deal with the mails OR
  • a post exit process (in cases where the pre-exit process was not carried out )- where the specific service the employee worked for, Legal, HR and internal audit would all review the account.  The specific service would look for e-mails the service needed to carry on with the employees work; Legal would look for e-mails needed for possible legal claims, contracts or agreements; HR would look for e-mails needed for possible grievance or disciplinary issues; Internal audit would look for any illegality

The approaches described by Rebecca and Lawrence are similar in two respects:

  • both approaches reflect a belief that colleagues can not be relied upon to comprehensively and routinely deal with individual e-mails as they go along by filing and deleting
  • both approaches  rely on a big effort just before or after  the member of staff leaves to deal with what is left in the e-mail account.  This is problematic.   All of our experience as records managers tells us that it is very hard to deal with backlogs.   E-mail communications are exchanged with such frequency that backlogs quickly scale up to a size that makes patient sifting and sorting impossible.  An e-mail account at the end of a person’s employment is in effect a filing backlog.

The only difference between the two approaches is that:

  • Rebecca’s organisation cannot guarantee that  the line manager /designated person of the departed staff member will review the e-mail content thoroughly, and move important mails to a more appropriate, more accessible place.  As a result they keep all the e-mail accounts as a back up, just in case there is an overriding need (legal or investigative) to find an e-mail from an ex member of staff.
  • Lawrence’s approach requires organisations to ‘feel the fear and do it anyway’.   There is still no guarantee that reviews have been carried out/carried out properly,  but this time the organisation presses the delete button after three months regardless.

Is it possible to deal with e-mail by asking staff to move important e-mails into an electronic or paper file as they go along?

Simon McCauley responded to Rebecca’s posting by saying that in his organisation  staff are expected to save important e-mails into the electronic document and records management system (Livelink) as they go along.

Simon’s organisation are planning to implement a policy of moving e-mails from people’s e-mail accounts to an e-mail archive six months after the date of the e-mail, then deleting them from the archive after a further twelve months.

I assume that the thinking behind such a policy is that:

  • they have confidence in the capacity of their colleagues to file important e-mails as they go along
  • they know that colleagues are much less likely to file as they go along if they  have the comfort of knowing that the e-mails are kept for them in their e-mail account anyway

The  State Records Authority of New South Wales (NSW) has given similar advice to NSW public offices.   They summarise their policy as follows:

State Records advises NSW public offices to capture email messages that are sent or received in the course of official business into a corporate recordkeeping system. State Records suggests two principle methods for capturing messages:

– capturing messages into an EDRMS (electronic document and records management system)

– printing messages and capturing them on paper files

In her blogpost reporting the findings of their  recent survey of  e-mail management in NSW public offices,   Emma Harris of State records reported that:

– 81% of public offices agreed with the statement that in their offices ‘e-mail messages with corporate value are stored only in personal email accounts and are therefore at risk of loss or premature destruction’

– 33% of respondents advised that employees in their organisation neither capture messages to an EDRMS nor print and file them.

– few organisations have investigated alternative approaches to managing e-mails’[as opposed to asking colleagues to move e-mails into EDRMS/print to file].

The blogpost went on to report:

– half of the responding organisations have implemented an archiving solution, with two products (Symantec Enterprise Vault and Quest Archives Manager) being the most commonly implemented.

– A number of email archiving solutions have retention and disposal functionality (e.g. the ability to set retention periods and disposal actions on messages and to destroy messages when retention periods have expired). However the results of the survey suggest that organisations with email archiving solutions are not actively managing the retention and disposal of messages using this functionality.

The findings betray a lack of confidence on the part of the NSW public offices in the adherence of their staff to the policy of moving e-mails to electronic or paper files. This lack of confidence is presumably what lays behind the fact that NSW are, like Rebecca’s organisation, keeping e-mail accounts indefinitely.

Can we still set a blanket retention rule on e-mail accounts if we know they contain important messages that we need as records?

There is a similarity between all four approaches – Lawrence’s, Rebecca’s, Simon’s and the New South Wales approach.  All four are based on moving e-mails out of e-mail accounts.

If, like Lawrence and Simon, we are confident that we can move important e-mails out of e-mail accounts, then setting a blanket retention period on those accounts not a problem.  We set a blanket retention period covering all accounts, and we make it as short as we possibly can to concentrate peoples minds

But what if, like Rebecca’s organisation, like New South Wales public offices, and like most of the organisations I have worked with and spoken to over the last decade, you are not confident that important e-mails are being moved out of e-mail accounts?   Then setting a retention period is a different type of exercise.  All of a sudden we are having to recognise that the e-mail account is a record – a record of the work correspondence of that member of staff.

A blanket retention period, however short or however long, is not appropriate for organisations whose e-mail accounts contain important correspondence that is not available elsewhere.   This is because the roles people play in organisations vary greatly in their significance and impact – you are unlikely to need a record of the correspondence of an accounts clerk in your finance department for the same length of time as the correspondence of your chief executive (with all due respect to both parties).

We need to find a rationale on which to base a retention rule on e-mail accounts.   This is something we as a profession have not hitherto thought through for the simple reason that we have been battling for over a decade to avoid having to treat e-mail accounts as records.  Even starting to think through the consequences of treating e-mail accounts as records feels like an admission of defeat.  In reality this is not an admission of defeat.  Defeat would come up if we gave up trying to keep manageable records of people’s work correspondence.

Getting people to move individual e-mails one-by-one to electronic files is a tactic not an end in itself.   Most organisations have not been able to make that tactic work – at the very least we need an alternative.

Establishing a defensible rationale for retention rules on e-mail accounts that we treat as records

We can set a retention period for a record of a particular type of work by considering all the different reasons why we need a record of the work in question, and then keeping  the record for the longest period that any of those needs is likely to stay valid.

The  e-mail account of an ex member of staff is simply a record of the correspondence exchanged by a particular individual in the course of their work, minus any e-mails that have been deleted/moved.

There are multiple legitimate reasons why someone might need to look at the work correspondence of a colleague or  predecessor who has left :

  • They might need to see what correspondence their colleague/predecessor had exchanged with a particular external stakeholder/partner/customer/supplier/citizen in order to inform their continuation of that relationship.
  • They might need to see what correspondence the colleague/predecessor had exchanged in the course of a piece of work because they need to continue with the piece of work. restart it,  learn from it, evaluate it, copy from it etc.
  • They might need to account for their colleague/predecessor’s work, in response to audit, investigation, criticism, access to information request or legal discovery
  • Depending on the nature of the role of that individual, they might need to transfer the correspondence to a historical archive on account of the enduring public interest in the work of that individual

In most parts of most organisations we cannot adequately meet those record keeping needs without retaining the e-mail account of the member of staff concerned.   The challenge of setting a retention value on e-mail accounts is that such accounts will typically contain corresondence arising from many different pieces of work, and  those pieces of work may have very different retention values.

A nice, neat approach is simply to keep the e-mails of an individual for as long as you keep the records of the main type of work that they carried out.

  • If they were an accounts clerk in a finance department, and your organisation’s retention rule on accounting work is to delete the records after seven years, then apply that rule to their e-mail account also
  • If they were a senior civil servant working on policy issues and on new legislation,  and your retention rule for work on the development of legislation, and on the development of national policy, states that records should be kept for  for 20 years and then reviewed for permanent preservation and transfer to a historical archives,  then apply that rule to  their  e-mail account also
  • If they worked on staff recruitment, and the retention rules for recruitment work is to delete records three years after the recruitment exercise,  then retain their e-mails for three years too.

One choice to make is whether to have the retention rule:

  • applied to the entire e-mail account – so the retention rule is triggered from the moment of the individual’s departure from the organisation (this has the disadvantage that some staff may have had long and varied careers in the organisation)
  • applied to e-mails by date (month or year)  –  so the retention rule is triggered by the end of the month or year that the e-mail was sent/received in (a better option)

The problem of personal data of a sensitive nature in e-mail accounts

So far so good – we have a defensible logic to base our  retention rules on e-mail accounts, to meet the full range of records management needs.  But there is a problem.  The problem is the widespread presence of personal data of a sensitive nature in e-mail accounts.  By ‘sensitive nature‘   I mean

  • information about the e-mail account holder that they would not want even their closest colleagues or their successor to access; and
  • information about a third party that the e-mail account holder corresponded with, or had discussed in e-mails, where that person could be disadvantaged if the information were to be made available even just to the account holder’s successor and closest colleagues

Even if an individual never used their work e-mail account for non-work correspondence with friends and family, their account is still likely to contain personal information of a sensitive nature, exchanged with colleagues.  Think of an e-mail exchange between a line manager and a member of their team who had to take time of work for personal or family reasons.

The fact that most e-mail accounts have not had such e-mails filtered out means that most organisations in my experience (centred around the UK and Europe) cannot currently allow colleagues routine access to the e-mail accounts of their predecessor, or their former colleagues.

Most organisations struggle to set access rules on e-mail accounts

Most electronic document management systems work on the principle that access permissions can be set for objects or aggregations of objects (file/folder/site/library/document etc.).   A person or group of people is either permitted or forbidden to access that object/aggregation.   There are no grey areas in between.  If I  am authorised to see a document then the system merely asks me to authenticate myself (so the system knows it is indeed me who is asking) .   It does not ask me why I want to see it.

Rebecca’s organisation allows access to archived e-mail on ‘a case-by case’ basis.  In other words they are unable to tell their e-mail archiving tool who is authorised  to access each e-mail account.

With e-mail archives the information contained in the archive is so sensitive that organisations are imposing an extra control – people are having to say why they need to access the e-mail account, and that request is either permitted or denied, not by the e-mail archive itself, but by people in the department responsible for overseeing the archive.

I worked with one organisation where any application to see e-mail accounts of former staff had to be approved by their human resources (HR) department, who would only allow consultation in exceptional circumstances where there was no other way of getting the information.   One  individual told me that any that they had wanted to access the correspondence that a former colleague had exchanged with a supplier about a particular contract, but HR had refused.

That HR department had no option but to be restrictive.  Imagine this scenario:  I work with a colleague, and  develop malicious intent, or an unhealthy curiosity, towards them.  They leave.  I think of a project that they worked on and say to the IT department that I need to look through their e-mail account to find records relating to that project.  What else might I look for/find?  That is why governance of e-mail archives is vital , including keeping non- deletable records of who searched for what terms under what authority , and what e-mails they opened and looked at.  This must include any searches made by any staff, whether end users or IT system administrators.

Is there any point in setting a retention rule that covers all the record keeping needs arising from an e-mail account if we cannot allow colleagues to access the e-mail accounts for those purposes?

The retention rule that we arrived at above was based on the full range of recordkeeping needs that we have in relation to the correspondence of an individual who is a close colleague or predecessor.  We now find that we cannot allow access to the e-mail accounts, even to close colleagues, for most of these purposes, because of the presence of personal information of a sensitive nature that is unmarked, unflagged, and undifferentiated from the rest of the mails in the e-mail account.

If we can only access e-mail accounts in response to overriding imperatives such as access to information requests, e-discovery requests and the need to defend or prosecute any legal case we might be involved in,   then should that be the only consideration we take into account in setting our retention rule? Should we only retain e-mail accounts for the period in which it is useful for us to have them in case of legal dispute?

If we only take into account the overriding imperatives of legal disputes and access to information requests then the logic for setting a retention rule becomes much more arbitrary:

  • if we adjudge the cost/risk of the e-mail accounts being subject to an access to information/e-discovery requests to be greater than the benefit of being able to use the e-mail accounts to support any case we would need to make in court,  then we would impose  a short retention period – perhaps the three months that Lawrence suggested
  • if we adjudge the benefit of being able to use the e-mail accounts of former members of staff to support any legal case we might want or need to make to be greater than the cost/risk of servicing access to information and e-discovery requests then we are likely to set a retention rule equivalent to a standard limitation period of seven years as Simon suggested (though you need to be careful with limitation periods – in some cases the clock of a limitation period may not start ticking until well after a member of staff leaves – for example if the person was working on designing a bridge, or a drug, or with children etc.)

The problem with this very pragmatic approach is that we will continue to fail to meet the day-to-day record keeping needs of our colleagues when they start a new job, and when they need to look back at the work of former colleagues.   And we will not not be able to make the record of the work correspondence of people playing important roles in society available to  future generations of policy makers, researchers and historians.

In his excellent Digital Preservation Coalition Technology Watch Report   on e-mail  Christopher Prom reported:

Winton Solberg, an eminent historian of American higher education, remarked … ‘historical research will be absolutely impossible in the future unless your profession finds a way to save email’ (Technology Watch Report 11-01: Preserving Email [PDF 916KB] by Christopher J Prom 2011,  page 5)

I will go one further and say that if we could solve the challenge of how we  provide an individual with routine access to the e-mail account of their predecessor, then we will be able to solve the challenge of how we provide access to that an e-mail account to historians or other researchers further down the line.  The two challenges are inextricably linked.

Many of our organisations have e-mail archiving tools, but these archives function as a murky sub-concious of the organisation, full of toxic secrets, inaccessible to the organisation in its normal day to day functioning,  and they pose a huge, ongoing,  information governance risk.

What we need is an approach to e-mail that results in staff leaving behind an e-mail account that their colleagues and successor can routinely access and use, without unduly harming either the account holder or people mentioned in their correspondence; and that we as an organisation can apply defensible access rules and retention rules to.

It is beyond the ability of a single organisation to develop such an approach (because it involves changes to available tools, changes to the way we think of an e-mail account, and changes to how we ask our colleagues to treat e-mail).  But it is well within the capability of the records management/archives professions to articulate such an approach, and then incentivise and cajole  venders (particularly the ecosystem around the big on-premise and cloud e-mail products/services) to create offerings that match it.

As a starting point I would like to see us as records managers and archivists getting this issue on the agenda of our organisations and of society more widely.

Two quick suggestions to get the ball rolling:

  • For records managers –  if you are concerned that important e-mails are not being moved out of e-mail accounts,  consider broaching the emotive subject of e-mail accounts when  building or revising your organisation’s records retention schedule.  Include in the retention schedule a list of those post holders in your organisation whose e-mail account contents you require be retained for a minimum of 20 years
  • For archivists working for the national archives of our nations -if you are concerned that important e-mails in government departments/ministries in your country are not being moved out of e-mail accounts,  then when you draw up or revise your selection policies,  include a list of posts in the various government bodies from which you require e-mail account contents to be appraised for permanent preservation in your archives