Why were corporate wide records systems in the paper age so much more successful than those in the digital age?

Records managers are often accused of trying to replicate ‘a paper paradigm’ in the digital world.  This is a little ironic. If we were able to implement corporate wide electronic records management systems that were half as good as the best records systems in the paper days then we would be very popular indeed.

Our predecessors in the records management profession 20 years ago  tried to abstract the qualities of the best paper records management systems and express these qualities as a set of technology neutral criteria, in the hope and expectation that we would be able to design electronic records mangaement systems that also met those criteria, even if they met them in a completely different way.

The best statement of these technologically neutral criteria can be found in  section 8.2.2. the ISO 15489 records management standard, (see my last post for a more detailed discussion of them)

The five characteristics are as follows. In order to be considered reliable a records system must:

  •  routinely and comprehensively capture all records arising from the activities that it covers
  • act as the main source of reference for the activities it covers
  • link records to the activities from which they arose
  • protect records from amendment or deletion
  • preserve access to records over time

These characteristics may at first sight seem utopian. No organisation I know of currently operates a corporate wide system that meets all these characteristics and covers all of their activities. And yet in at the time they were drawn up, in the early 1990s,  they seemed anything but idealistic. Before the introduction of e-mail, any organisation that wished to could set up a record systems that met all five of these characteristics.

What made it possible to set up a reliable corporate wide records system in the paper age?

In the paper age there was a gap in time and space between:

  • the point in time at which a business document/communication arrived in the organisation from outside AND
  • the point in time at which that business document/communication arrived in the in-tray of the individual responsible for dealing with that communication

There was also a gap in time and space between

  •  the point at which an individual within the organisation sent a business document/communication out AND
  • the point at which that communication either arrived at the colleague it was addressed to, or left the organisation for dispatch to an external recipient

Organisations could insert control points into those gaps in time and space to ensure that business communications were routinely captured into the records system, and assigned to containers (usually called ‘files’) that each represented an instance of a particular activity.


How a registry system typically worked


The illustration above shows how such a registry system would typically work

Incoming post would arrive in a post room. The post room staff would do a rough filter of the post

  •  Things that looked like they were not needed for the record system because they were trivial, personal, or reference material (promotional material/flyers/postcards/love letters/magazine subscriptions) would be sent direct to the individual concerned
  • Things that looked like business communications (letters, memoranda, reports etc.) would not be sent direct to the addressee, instead they would be sent to the relevant records registry

Each registry was simply a team of records clerks who looked after the files for the area of the organisation within the scope of their registry. Typically an organisation would have several registries, each covering one or more of the organisationsfunctions/departments/buildings, though it was also possible to operate with one central registry covering all activities.

The registry would assign the document to the file representing the activity from which the correspondence arose. They would deliver the file, with the new correspondence on it, to the action officer.

The action officer would draft a reply which would be typed up by a typist in a typing pool. The typist would create two copies for the action officer to sign – one to go on the file, and one to be sent out.


Evaluating registry systems against the five criteria for a reliable records system

The registry system described above meets the five reliability criteria for a records system because:

  • There is routine and comprehensive capture into the record system.   In the post room(s) the same staff filter incoming correspondence day after day. They apply the same thought process to post day after day to decide what post goes to which registry, and what post bypasses the registries and gets sent direct to individuals. If the post room acquires a new member of staff, they train that person in the thought process.     Similarly in the registry(ies) the same staff do the filing day after day.   The registry staff have no interest in withholding embarrassing material from the file. The file is not holding them to account, it is holding action officers to account. The files are comprehensive – every incoming  piece of correspondence goes through the post room, and is either filtered out or sent to a registry for filing.
  • The colleagues working on the project/case/relationship/matter use the ‘file’ as their source of reference. If there are gaps in the file a colleague is not only likely to notice, there is also a fair chance that they will be motivated to do something about it, because they rely on that file being complete in order to be able to do their work, and defend their work.
  • A file is set up every time a new piece of work starts and every piece of correspondence placed on the file is, by the act of being placed on a file, connected to the activity it arose from.
  • The registry guarded the files. They typically kept a record of who each file was loaned out to and when. It is true that there was nothing to physically prevent an action officer ordering up a file, removing a paper that was incriminating to them from the file, and returning the file. However they would risk dismissal if detected.
  • Organisations had ways of managing the records lifecycle so that access to records were preserved over time. The registries would store active files close to office space, then send records to a record centre when they became non-active at a point some time after the work had finished From the records centre records would be disposed of at the end of a designated retention schedule either through destruction or through transfer to an historical archive

What happened to these systems when e-mail and networked computers arrived?

On the face of it it seems that the records management and archives profession was in a good position circa 1993 to ensure that no paper registry system was decommissioned without an adequate digital replacement being put in-place. Those organisations that operated such systems tended to proud of the systems, and proud of the records that the systems held. Their records were their support, their defence and their source of reference. They had no plans to jeopardise the quality of their records.

So why did that not transfer?  Why is it that even organsiations that had great records systems could not replicate the quality of those record systems after e-mail?

There are three main reasons to this:

  •  the speed and manner in which an e-mail moves through space and time is so different from that of a piece of paper that even having abstracted the qualities of a good paper records system into a set of criteria it was hard for the profession to imagine a way in which a system in the post e-mail world would meet those criteria
  • individual e-mail accounts collapsed the  time and space between a piece of correspondence arriving in an organisation and it arriving at the desktop of the action officer.    There was no time or space for records management controls to be inserted
  • the rapid and uniform spread of e-mail, through standard e-mail client and server software, meant that organisations denied themselves the opportunity to innovate when they set up their systems for handling e-mail.   A satisfactory method for transparently filtering and classifying/ filing e-mail never emerged because so little experimentation was done.


Evaluating automated approaches against records management principles

As a profession we are very proud of our principles.  If you ever discuss technology with a group of records managers one of us is bound to say  ‘records management principles are timeless, regardless of how much or how quickly technology changes’.

But what exactly are these principles?

The time when you most need a set of principles is when new ideas, tools, technologies or approaches come onto being.   By definition we have little or no practical experience of these new ideas/tools/ technologies.  We need a set of principles which distills our past experience of what has and hasn’t worked, in order to predict whether these new approaches are likely to work.

NARA’s report on automated approaches to records management

In March 2014 the  US National Archives (NARA) issued a report on the different ways in which records management could be automated.   The approaches have little in common with each other except that they all aim reduce the burden of records management tasks on end users as compared with the more established electronic records management system approach.

The approaches discussed in the report were extremely varied, but can be boiled down to the following:

  • an in -place approach based on holding a records classification and related retention rules in one application, and applying them to content held in the various different native applications of the organisation (SharePoint, e-mail accounts, shared drives etc.)
  • a workflow approach –  the definition of workflows for each activity, that include provision for the capture of records at particular points of the process
  • a defensible disposition approach – the definition of policies to apply to aggregations of content around the organisation.    For e-mail accounts you might have a policy that the e-mail accounts of individuals deemed to have important roles in the organisation are kept permanently, whilst the e-mail account’s of less important individuals are kept for six years.  You might set a policy that content in SharePoint sites/shared drives is deleted after five years/fifteen years. or kept permanently depending on the importance of the functions carried out by the team concerned.   There is no retention schedule, and no records classification, just risk based decisions.
  • automated filing by a rules engine  through the definition of rules to enable a rules engine to recognise which content arises from which particular activity
  • automated filing by an auto-classification tool  the  use of machine learning to learn the attributes of content that arises from particular activites,  without the organisation having to write a rule set.

(I have taken the liberty of changing NARA’s categories slightly – this summary  sticks more closely to NARA’s definitions)

 These new approaches are very different from the precious electronic records management system approach, but there has been no change in records management theory in between the two approaches (unless you count the new ideas from the information governance world – such as ‘everything held is a record’  ‘big buckets are better than granular hierarchies’ ‘end-users should not have to bear the burden of records management’)

NARA gives the pros and cons of each automated approach, without favouring any one or other of them, and without stating whether or not they believe each approach will keep records to an acceptable standard on a corporate scale.    This is not NARA’s fault – it is simply a recognition of the fact that at the moment we as a profession have no handy set of criteria to evaluate these approaches against.

In this blogpost I am going to nominate what I think is the  most useful and concise set of criteria for judging a records management system or approach- namely the five characteristics of a reliable records system that were developed in Australia in the early 1990s and ended up as section  8.2.2 of  the International Records Management Standard ISO15489.

The five characteristics of a reliable records system

Reliability is the most important characteristic of a record system.  A record system exists to perform a paradoxical function.  It exists to both:

  •  enable external stakeholders to scrutinise an organisation AND
  •  to enable the organisation to defend itself against external scrutiny

This paradox is fractal – it is present at any level of aggregation:

  • a record system enables an organisation to scrutinise a team, and it enables the team to defend itself from scrutiny
  • it enables an individual to be scrutinised and to defend him/herself from scrutiny.

The only way that a records system can resolve this paradox is by being trusted by all parties – in other words for all parties to consider the system to be reliable  – the individuals carrying out a  piece of work, their immediate colleagues and line management, the rest of the organisation, and external stakeholders.

Section 8.2.2.of the ISO 15489 standard states that in order to be considered reliable a records system must:

  • routinely and comprehensively capture all records arising from the activities that it covers
  • act as the main source of reference for the activities it covers
  • link records to the activities from which they arose
  • protect records from amendment or deletion
  • preserve access to records over time


Why the five characteristics of reliability are essential, rather than merely desirable

These characteristics are not ‘nice to have’ they are ‘must have’.

Think what would happen if a records management system did not meet even one of these characteristics:

  • If records were not consistently captured into the record system then external stakeholders could wonder wether there was a bias in record capture, for example whether content that could be incriminating/embarrassing was deliberately kept off the record system
  • If a record system does not comprehensively capture all records arising from an activity then this will leave gaps in the record and weaken the organisation’s ability to defend or learn from the way it carried out that activity.  It will also lead to external stakeholders looking to other sources of information outside of the records system – sources that may be more time consuming and more embarrassing for the organisation to search.
  • if a record system does not serve as the main source of reference for the activities within its scope then any gaps in the record will go unnoticed, and uncorrected.   It is particularly important that the record system is used as source of reference by colleagues carrying out the piece of work itself – because they are the only people in a position to know that there is content missing from the record.
  • if records are not linked to the activity from which they arose then the organisation will find it impossible to apply a precise retention period to those records.   Retention periods are specific to a type of activity:  managing staff,  designing a building, managing a project, bringing a pharmaceutical product to market,  adjudicating on a citizen benefit claim etc..  The trigger point for that retention period to start is even more specific, it is specific to a particular instance of each activity:   the date a particular member of staff left employment;  the date a particular building ceased to exist; the date a particular project finished, the date a particular pharmaceutical product was withdrawn from the market, the date a particular citizen ceased receiving benefit etc.
  • If records are not protected from amendment or deletion then an external stakeholder will have cause to doubt whether or not content detrimental to the organisation, or to a particular team or individual, has been deleted from the system prior to them viewing the record
  • If access to records is not preserved over time then an organisation cannot be sure that if will be able to defend itself from scrutiny/challenge if that scrutiny or challenge comes at a date in the future.  By the same token external stakeholders cannot be sure that they will be able to scrutinise the organisation at a future date.


Why have we lost site of these reliabilty criteria?

At first site it seems odd that we as a profession have lost site of these criteria for judging a records system.

These criteria are not obscure.    They were a fundamental part of the drive by the records management and archives professions to manage the transition from the paper age to the networked digital age by expressing the attributes of good recordkeeping systems in an abstract, non-format specific way.

Although these criteria are trying to be as timeless as possible, they are also very much of their time.  They were written at the start of the 1990s, just before the mass adoption of e-mail and networked computers with shared drives  by organisations in what were then called developed economies. They were embedded in the Australian records management standard which later became the International Records Standard (ISO 15489, published in 2001).

The reason why we have lost site of them is that we have not been able to implement systems that meet all five of these criteria,  on the scale we want to work at (the corporate scale) since the mass adoption of e-mail.  Nor is there a realistic likelihood that any of the five automated approaches discussed by NARA will meet all five of these criteria.


Evaluating existing record keeping systems against the five reliability criteria

When you compare existing systems within organisations to these reliability criteria you find that:

  • line of business systems (case file systems and sector specific systems such as insurance claim systems) can be set up to routinely meet all five reliability criteria.   They can use functional e-mail addresses, and/or web forms, to divert correspondence related to that function away from private e-mail accounts directly into the system, which means they can routinely and comprehensively capture correspondence arising from  the activities that they cover.   However each line of business system can only cover one area of the business.    Organisations carry out  so many different types of work that it is impossible to have a line of business system for each of them.
  • generic document management repositories such as shared drives, electronic records management systems, and SharePoint cannot routinely and comprehensively capture business correspondence sent and received through e-mail – they are dependent on individuals exercising their own judgement on which e-mails (if any) they upload or push to the system.    None of these three types of repository serve as the main source of reference to all or most of the activities that they cover.    Individuals tend to use their own e-mail accounts as the first place  to go to find records of their work.   Of those three repositories electronic records management systems and, to a lesser extent SharePoint, do a better job than  shared drives at protecting records and linking them to the business activities that they arose from.
  • e-mail archives routinely and comprehensively capture electronic correspondence.  However they do not relate records to a business activity which leads to problems with applying access and retention rules .  They cannot be used as the main source of reference for activities by anyone except Legal Counsel because private and sensitive e-mails are undifferentiated from other correspondence


Evaluating automated approaches against the five reliability criteria

The automated approaches described in the NARA report  either don’t meet all five reliability criteria or they don’t scale across an organisation

  • the in -place approach – currently lacks an answer to the question of how  to routinely and comprehensively capture important e-mails and relate them to business activities.  It is dependent on either individuals dragging e-mails into folders (which is not routine or comprehensive because many individuals never file e-mail into folders) or one of the automated filing approaches (see below)
  • a workflow approach – works well, meets all five criteria, but cannot scale across an organisation because of the time taken to analyse processes and define workflows
  • a defensible disposition approach – does not protect records.   A disposition rule is applied to a part of a shared drive or a SharePoint site but indviduals can delete or amend content before the time at which the disposition rule is applied.
  • automated filing by a rules engine  – the time spent to write the rules means this approach does not scale to a whole organisation
  • automated filing by an auto-classification tool –  a typical organisation carries out a great many different types of activity.  For each type of activity it carries out a great many different instances of that activity.  An auto-classification tool  has to be trained in each container/category it is asked to file records into.   The more containers/categories you have the higher the cost of training.  Organisations trying to implement auto-classification corporate wide have had to compromise and ask the tool to file into ‘big buckets’ rather than into a container for each instance of each activity (project/case/matter/ relationship etc.).  This means that on a corporate scale auto-classification does not currently meet the criteria of linking records to the business activities that they arose from.  I explained above that although retention rules may be set on broad swathes of activites (e.g records of all our projects are kept for ten years after the project ends) to apply that rule you need to have each record allocated to a particular project so that the ten years can be triggered by the end date of that particular project.   The buckets also end up being too big to be navigable or useable by end users, meaning that the system ends up not being used as the main source of reference for the activities it relates to.



At this point in time we need to be honest and say that we have no approach to implementing systems on a corporate scale that will routinely and comprehensively capture business correspondence , protect it, link it to the business activity it arose from , maintain access to it over time, and  serve as the main source of reference for the activities it covers.  Nor have we the prospect of such an approach any time soon.

Until we get such an approach organisations will suffer problems with their records management/information governance.

We have seen organisations establish electronic records management systems/SharePoint implementations and hope that access to information requests/ e-discovery can be confined to those systems, only to find themselves searching e-mail accounts, shared drives and maybe even back up tapes.  This is because their official records system does not routinely and comprehensively capturing records.  Their external stakeholders have responed in effect by treating all the organisation’s applications as being part of their records system.

We need approaches to records management that result in systems that are both reliable and relied upon.   If end users do not rely upon a records system,  but instead refer mainly to content outside the scope of the record system, then they will neglect to point out omissions in the record system, and there will be a disconnect between the records available to the individuals carrying out the work, and the records available to those wishing to scrutinise or continue their work.