A new wave of tools has hit the records management space over the past two or three years:

• eDiscovery indexing engines that aim to index all of an organisation’s content across however many repositories/applications it uses
• in-place records management tools that aim to apply classification and retention rules to content regardless of the repository/application in which it is kept
• e-mail archive tools that are more ambitious than the previous generation of e-mail archives, and which, for example, offer features supporting the auto-classification of e-mail
• plug-ins for SharePoint to bridge the gaps in records management functionality
• clean-up tools for shared drives

All of these tools can be placed under the the collective description of ‘information governance tools’.  But they are very different from each other.

One way in which we can categorise these new tools is by the ambitions of the organisations deploying them:

• An organisation with a big eDiscovery bill but little records management ambition will turn to indexing engines to give them the reassurance that they can identify material responsive to litigation cases even if individual staff continue to leave correspondence in their e-mail accounts and do nothing more with documents than stick them in a folder on a shared drive.
• Organisations wanting to reduce the load of burden of maintaining (and reviewing when an eDiscovery/access to information request comes in) vast volumes of documentation on shared drives will deploy a shared drive plug in tool to protect and classify material of value, whilst identifying and getting rid of ROT (redundant, outdated and trivial) documentation.
• Organisations concerned about e-mail volumes, and  with the potential existence of toxic comments and information within e-mails, will deploy e-mail archiving tools. They will either use auto-classification features to filter e-mails into categories and apply disposition rules,  or they will use analytics features aimed at identifying  high risk/trivial/private communications .  The auto-classification tools are indeed getting more and more sophisticated but the applications of  auto-classification tools are still crude.  You need to train an auto-classification tool how to recognise material relevant to every single category in whatever classification you are using.  The more granular the classification the more training you have to give the auto-classification tool.  For the time being at least you can only realistically auto-classify into ‘big buckets’.
• Organisations with records management ambitions and with big investments in SharePoint will deploy SharePoint plugs ins.  The plug-ins will enable them to:  link retention rules to their records classification;  apply the classification and retention rules to different types of SharePoint objects:  (folders/content types/libraries/sites) ;  and better import and export content into and out of SharePoint.  The plug ins will also give them e-mail integration so that staff can drag and drop e-mail into SharePoint libraries.  The challenge here is that even a drag-and-drop facility does not on its own motivate end-users to consistently move significant material out of their e-mail accounts
• Organisations wanting to manage records across several different environments (typically shared drive, e-mail SharePoint), without asking staff to move records into a separate electronic records management system, will deploy in-place records management tools.  In effect these tools will give you much of what a  SharePoint plug-in tool and a shared drive clean-up/governance tool will do, and some of the things an e-mail archive might do. They will also offer connectors to the major enterprise content management system/document management system products on the market.   So long as the vendor of the tool remains viable and keeps coming up with connectors for new repositories you in theory have an approach to managing records across your whole IT estate.   The success of in-place vendors will depend upon the extent to which they can convince organisations that the synergy of having one tool to manage records across many applications outweighs the option to pick best-of-breed solutions for each particular application/repository (e-mail/shared drive/SharePoint).     The challenge for organisations is that each repository/application that they are trying to govern has its own unique features, structure and functionality.   This means that the in-place tool has to work in a different way to govern content in each separate application. The deployment of an in-place tool to each different application/repository is a separate project in its own right.

In practice we are already seeing convergence between these products as vendors either move into each others territory, or ally with each other.  We are also seeing convergence between these products and the previous generation of  electronic records management systems

• We are seeing vendors of indexing engines such as Zylab and Nuix moving more deeply into  information governance by adding functionality to  clean-up and apply rules to the repositories that they have indexed.
• We are seeing the vendors of traditional electronic records management systems such as IBM and HP Autonomy use their electronic records management systems as repositories  behind their in-place records management  offerings.  Even when an organisation adopts an in-place records management approach they are still going to want to decommission applications at some point and hence need to be able to move content out of those applications into a repository.
• We are seeing alliances between vendors of these different products – for example that between RSD and Nuix to offer both in-place records management and indexing/eDiscovery capabilities.

There are numerous  questions for us to explore concerning the implications of the rise of these tools are for records management.

• Is records management being subsumed into information governance – or is it a separate discipline that will help to shape information governance but will retain its own distinct identity and purpose?
• What are the fundamental differences between this wave of information governance tools, and the wave of ‘electronic document and records management systems’ that dominated the records management market between 1999 and 2009?
• Does this new wave of information governance form part of a wider change in records management paradigm? Are we seeing a new model of how records management should be tackled?    If so to what extent is it a fully worked through paradigm?  What are the underpinning set of beliefs behind it?   Is there a body of theory behind it?
• To what extent will such a new records management paradigm meet the aspirations of the profession?   Will it work in practice?  What would it need from organisations, from records managers, from end-users and from vendors in order to make it work?

The Food and Agriculture Organisation (FAO) of the United Nations (UN), with its HQ in Rome,  exists to ‘spearhead international efforts to defeat hunger and build a food-secure world for present and future generations’.

Last week I recorded a podcast with Ian Meldon, a records management consultant working for the FAO,  in which Ian described the approach of FAO to implementing a records management system based around e-mail.

FAO have had a system for managing electronic records for over a decade, and the system has always been based on e-mail.  They recently replaced their old system with a new approach.  The new approach involves:

• filtering out personal and trivial messages, so that significant messages can be managed and shared
• providing colleagues with ways of keeping abreast of the work e-mail traffic of team-mates, without those team mates having to copy each other into those e-mails.
• applying a records classification to significant e-mail messages without asking colleagues to interact with that corporate records classification

The previous records management system at FAO

From the year 2000 FAO had asked colleagues to copy or forward any e-mail needed as a record to the e-mail address of their local registry, where registry staff would file the e-mail in a Microsoft Outlook shared folder structure.    The system worked tolerably well, although compliance with the policy varied from area to area.

One weakness of the previous system was that all the records were kept within the Microsoft Exchange environment. People could only see the records of their local area – there was no possibility of a FAO wide search.  There was no sustainable way of holding and applying retention rules to the records.

Principles behind FAO’s new records management system

When FAO decided to overhaul the records system they based their approach on three principles:

• Don’t appear to introduce a yet another computer system  FAO have procured and implemented a robust electronic records management system (Filenet from IBM), for use as their repository.  But end-users never need interact directly with the Filenet repository – everything they need to do on the system can be done through the Outlook e-mail client.

• Don’t ask people to do something they are not already doing The idea was not to ask users to do anything more time consuming than the previous system’s demand that they copy in the registry to significant e-mails.  Under the new system every time a colleague sends an e-mail, a records capture pop-up appears asking them to say whether the e-mail is either a) personal or trivial or b) draft/transitory or c) FAO record.   If an individual selects personal or trivial then the e-mail is sent without going into the records repository.  If the individual selects either draft/transitory or FAO record then they are asked to choose the appropriate ‘team tag’  for the message (the team tag denotes which team they were working for in sending the message). The message then gets sent and a copy is placed in the records repository.  There is also the opportunity to mark a message as confidential if it is work related but there is a need to restrict access to it.

• Provide something useful beyond the need to keep records At 10pm every night the system generates a ‘digest’ for each team tag.  The digest is an e-mail that lists and links to all the FAO Record e-mails sent that day and tagged with that team tag.  This means that each morning an individual can see at a glance all the significant e-mails sent by colleagues in their team the previous day.  This has reduced the need for colleagues to ‘copy each other in’ to e-mails.  Furthermore individuals can choose to receive digests from other teams (if they have appropriate permissions).  If a manager oversees six or seven teams they can look at the digest for the six or seven team tags each morning, without needing to be copied into hundreds of e-mails.

The illustration above shows the team tag digest e-mail generated by the records system for the team tag ‘IPA-RMMP (the Records Management Modernisation Project)  on 17 February 2012.   Members of the team, plus anyone who had decided to subscribe to that team tag, would have received that digest late on the 17 February.  It gives them the subject line and first line of each e-mail.  It is presented in reverse-chronological order by time sent.   The digest is simply an automatically generated search query.   Any colleague can search the records repository from their Outlook client and generate a similar report, showing the FAO Record e-mails of any team over any time period, provided only that they have appropriate permissions for that team tag.    

The nature of the team tags

FAO created team tags by simply asking every area of the organisation to identify what teams they had, and who worked in those teams.

Team tags are maintained by registry/records management staff who create new team tags when new teams or project teams emerge, assign individuals to membership of particular team tags, and maintain the access permissions around team tags.

An individual might belong to one, two or several teams.  When they send an e-mail from the Outlook client the records capture pop-up asks them to assign a team tag to it if they have marked it as draft/transitory or FAO record.  

The pop-up presents them with a drop-down list of the teams they are assigned to.   If the individual is assigned to only one team then they have no choice to make – the team tag will be filled in automatically.

Application of the record classification

FAO created a records classification based on a functional analysis of the activities of their organisation.

The challenge was how to apply the records classification to the records that would build up in the system.   If FAO had asked individual users to place each e-mail into a file within that records classification then it would have broken their principle of not asking people to do something they were not already doing (in FAO’s previous system registry staff had placed e-mails on files on behalf of end-users).

Experiments with auto-classification

The first approach FAO trialled was auto-classifcation, where an auto-classification tool would allocate e-mails declared as  Draft/transitory and FAO Record  to the appropriate functional classification.

Daniel Oliveira  who worked on the project with Ian, told me that the auto classification worked amazingly well in areas such as the finance where the subject of messages were relatively consistent and predictable,  but it did not work nearly as well in the policy areas where the subjects of messages were unpredictable and unrepeated.   Policy work constitutes a significant proportion of FAO’s work.

Mapping team tags into the functional records classification

The approach they settled on was simply to map the team tags into the functional records classification.   Each team tag is linked to one node in the records classification.

FAO are building up in their repository what is in effect a correspondence record for each team, sortable by sender, recipient and date.   Each of these correspondence records is linked to the functional records classification, from which it can inherit a retention rule.

To my eyes they have struck a neat balance between the strongly individual centric nature of e-mail as it has emerged over the past two decades, and the more collective tradition of files and record keeping.

Access permissions on e-mail

E-mail saved as ‘draft/transitory‘ and ‘FAO Record‘ enters the records repository (FileNet) and inherits its access permission from the team tag, unless it had also been marked as Confidential.  FAO encourages teams wherever possible/appropriate to authorise all FAO staff to access the e-mails tagged with their team tag.

Teams are able to set a different access permission for ”draft/transitory‘ than for ‘FAO Record‘ if they wish to make a distinction.

How FAO deals with incoming e-mail

When FAO colleagues go to send an e-mail,   a records capture pop-up intervenes to prompt them to capture the e-mail to the records repository if it has some significance.

However when colleagues receive an e-mail there is no such opportunity to intervene with a pop-up.

FAO provide staff with two alternative ways of capturing incoming e-mails as records:

• Treating an incoming e-mail in the same way as any reply made to it –  when a colleague goes to send a reply to the e-mail the records capture pop-up intervenes – if they indicate that the reply is FAO Record (or draft/transitory) then not only the reply, but also the incoming e-mail that prompted it, will be captured into the repository

• Right click menu option – colleagues can select a message in their inbox and use the right click menu to capture it into the record repository and give it a team tag

How FAO deal with e-mail sent from mobile devices

There is a trend for colleagues to access e-mail via mobile devices such as smartphones and tablets. These devices do not have the extended Outlook client with the FAO records capture pop up.  There are too many varieties of smartphones/tablets out there for it be feasible for  FAO to develop a customised  e-mail client for each device.

FAO have given staff a generic e-mail address that they can copy e-mails sent from their mobile devices into.   Staff working in the Registries capture those e-mails into the system and give them the appropriate team tag.

Searching for e-mail

The search facility for the records system is built into the Outlook e-mail client.

To my eyes the search looks more intuitive than the advanced search in a typical electronic records management system.  This is because the metadata of e-mail (from/to/cc/subject/date) is simpler, more standardised and more intuitive than the metadata collected from end users in document profiles by electronic management systems.

This search allows people to specify a date range they are interested in, and to generate what is in effect a report of all the working e-mail sent/received by a particular individual or team within that date range (assuming they are in the appropriate permissions group)

Conclusions

There are six aspects to the FAO approach  that I find particularly valuable and interesting:

• They haven’t tried to fight e-mail they have not tried to get users to move e-mail out of the e-mail environment and into an environment that works on a completely different logic (the logic of a folder structure/ fileplan hierarchy)

• They have used the strengths of e-mail  – the fact that you can intuitively search and/or sort e-mail  by sender/date/recipient and title;  the fact that you have the context of who a document/messages was communicated to and when; the fact that colleagues spend most of their computer time in the e-mail environment; the fact that any document of any significance in environments such as the shared drive will pass through e-mail

• They have mitigated the main weakness of e-mail  – the fact that trivial and personal messages sit cheek-by-jowl with work messages and make it problematic to provide access to the e-mail of colleagues

• They have paid as much attention into making sure that the records  function as a useful information and news source for colleagues, as they have paid to making sure people contribute records to the system

• Their approach does not depend on any particular proprietary software.  They have used a proprietary electronic records management system as their repository for the e-mail, but  they could have used any one of a number of different electronic records management systems and achieved similar results.  The customisation of the e-mail environment was done by their in-house development team.  Their approach does not depend on clever algorithms or sophisticated auto-classification rule engines.

• They have moved away from files and filing,  but still group records into meaningful and manageable aggregations  Records are accumulating in manageable aggregations, but these aggregations are slightly different from files we have been used to in old paper filing systems, and the files that we created in  electronic document and records management systems.  Those files attempted to capture the whole story of each particular piece of work.   They depended on teams setting up a new file for every new piece of work that they started.   The nearest equivalent of the file in the FAO set up is the team tag.  But teams have not been required to create/request a new team tag every time they start a new piece of work – they keep the same team tag for all the work they undertake.  The record is in effect a correspondence record of that particular team over a particular time.  This is less granular than a traditional file structure.  But the loss of granularity is compensated for by the ability to sort on sender/recipient and date.

The dream of a single record keeping profession

The continuing divide between records managers and archivists

• The records managers refer to MoReq2010 (developed by the DLM Forum itself ) –  a specification  of the functionality that an application needs in order to manage the records it holds
• The archivists talk about OAIS (open archival information information systems) a standard for ensuring that a digital repository can ingest, preserve, and provide renditions of electronic records that have been transferred to the archive

Records management initiatives

• building in records management functionality into business applications (and storing records in those applications) (in-place records management)
• storing records in the business applications into which they were first captured, but managing and governing them from a central place (federated records management)
• integrating the many and various business applications in the organisation into a central repository which stores, manages and governs records.

Archival initiatives

• building a digital archive that can receive accessions of electronic records and their associated metadata
• defining a standard export schema/metadata schema dictacting exactly how what metadata needs to be provided, and in what form, about records transfered to the archives
• enforcing that export schema/metadata schema so that all new transfers of electronic records come in a standard form that is relatively straightforward for the archive to accession into their digital repository

Lack of a join between records management and archival initiatives

The OAIS model

• The object originally transferred to the archive is a submission information package (SIP)
• The object stored in the archive is an archival information package (AIP)
• the object provided to the requestor is a dissemination information package (DIP)

An example of the OAIS model way working well – The Danish National Archives

An example of the records management approach working well – the European Commission

How do we make it more feasible to manage records over their whole lifecycle?

• 
implement applications that keep and records and associated metadata in that standard format, OR
• 
 implement applications that can export metadata in the standard export format , even if the metadata within the application had been kept in a different way
• develop the capability to transform exports from any of their applications into the standard export schema. This last point should be helped by the fact that any widely accepted export schema would lead to the growth of  an ecosystem of suppliers with expertise in converting exports of records and metadata into that format.  Indeed such a format could become a ‘lingua franca’ between different applications.

The opportunity for a link between MoReq2010 and the OAIS model

Debate at the DLM members forum on an OAIS compliant extension module for MoReq2010

Alison Gibney spoke to the June meeting of the IRMS London Group about the relationship between the disciplines of information assurance and records management.

Alison differentiated  information assurance from information security.  Information security covers any type of information an organisation wants to protect, whereas information assurance is focused on protecting personal data.    There is no US equivalent term, partly because the US legislation on personal data is less strict than that of the UK.

Alison said that in the UK public sector over the last five years records management has gone down a peg or two (thanks to budget cuts) , whilst  ‘information assurance’ has gone up a few pegs.  The rise of information assurance is thanks to the various high profile central government data leaks in 2007 and 2008; the Hannigan report which compelled UK government bodies to adopt a strong information assurance regime in response to those leaks; and the fines meted out by the Information Commissioner for non-compliance with the Data Protection Act.

Alison showed a list of the fines meted out by the Information Commissioner over the past few years.  She pointed out that a significant proportion of the fines had gone to local authorities.  This was not necessarily because local authorities are worse at managing personal data than, say, a private sector retail company.  It is more likely to be because local authority have literally hundreds of different functions that necessitate the collection, storing and sharing of personal data, whereas a retail operation may only have three or four such functions.  It is very hard for a local authority to ensure that all these many different functions are fully compliant with data protection legislation.

Alison also pointed out that most of the fines could be ascribed to two generic types of breaches:

• communications being sent to the wrong person
• loss or theft of removable media

These two types of generic breaches both occurred across a range of formats, digital and hard copy.   Communication misdirections included misdirected e-mail, letters and faxes.  Loss and thefts of removable media included losses of laptops, key drives and hard copy files.

When to encrypt and when not to encrypt

Alison said that encryption offered a solution to the problem of protecting personal data in certain circumstances.
Alison recommended encrypting:

• personal data in transit (for example data being e-mailed to a third party) because of the risk of interception or misdirection
• personal data on removable media (optical disks, laptops, mobile phones etc) because of the risks of loss or  theft

Alison did not recommend encrypting ‘data at the rest’ in the organisation’s databases/document management systems.   But she raised a question mark over data in the cloud.  Technically it is data at rest.  But it is data held by another organisation, possibly within a different legal jurisdiction, and the organisation may wish to encrypt because of that.

It is worth taking a closer look at the issues around the encryption of the different types of data that Alison mentioned.

Encrypting data in transit – e-mail

There are various ways of encrypting e-mail.   A typical work e-mail travels from a device, through an e-mail server within the organisation, to an e-mail server in the recipient’s organsiation, to the device of the recipient.  There are security vulnerables at any of those points.  Devices bring in a particular vulnerability particularly since the rise in usage of smartphones and of trends such as ‘bring your own device’.

The most secure option is for the message to be encrypted all along the chain.   However the further along the chain the e-mail is encrypted the more complex, expensive and intrusive the encryption software and the procedures for applying it become.   Chapter 4 of this pdf hosted by Symantec gives a neat summary of the different options for e-mail encryption.

If you decide to encrypt messages from the moment they leave the sender’s device  all the way to the recipient’s device (endpoint to endpoint) then both the sender and the recipient will need encryption software installed on their device.  This is intrusive for both parties.  It may be reasonable to expect organisations that you regularly exchange sensitive data with to have such software installed.  It would not  be reasonable for a local authority corresponding with a citizen for the first time to expect the citizen to install such software.

A less intrusive option  is ‘gateway to gateway’ encryption.  The  message goes in plaintext from the sender’s device to a gateway server inside their organisation. The gateway server encrypts it.  When it reaches the recipient organisation it is decrypted by a gateway server which sends it on in plaintext to the recipient.  Note that this model requires the recipient organisation to have the same encryption software installed on their gateway server as is used by the sending organisation.

A lighter approach to encryption is the gateway-to-web approach where a standard plaintext e-mail is sent to a recipient giving them a link to a web address where they can go to retrieve the message which is protected by some kind of transport layer encryption such as SSL (Secure sockets layer – as used to  protect credit card transactions on the web).  The web site will ask the user for authentication.  Assuming that the authentication details have been sent to the user by a different channel other than e-mail, this will provide some protection against an e-mail being sent to the wrong recipient.  In her talk Alison had informed us that 17 London boroughs had chosen encryption software from one particular vendor (Egress) – that uses this gateway- to-web model.  Although this is a lower level of security than endpoint to endpoint encryption, it has the crucial advantage that the recipient does not need to install encryption software.

Encrypting data on removable media

Encrypting personal data on removable devices is the most straightforward of the cases that Alison mentioned.  The individual who owns the device it with their (public) encryption key.  Only they have the (private) encryption key necessary to decrypt it.  If the device gets lost or stolen the data is safe so long as the person who gets it does not get the decryption key.

Encryption and data-at-rest

Bruce Schneier points out in his post Data at Rest vs Data in Motion (http://www.schneier.com/blog/archives/2010/06/data_at_rest_vs.html) that cryptography was developed to protect data-in-motion (military communications), not data-at-rest.   If an organisation encrypts the data it holds on its on-premise systems then it faces the challenge of maintaining through time the encryption keys necessary to decrypt the data. Schneier puts it succintly:  ‘Any encryption keys must exist as long as the encrypted data exists. And storing those keys becomes as important as storing the unencrypted data was’ .  If you are encrypting the data to guard against an unauthorised person overcoming the system’s security model and gaining access to the data, then logically you cannot store the decryption keys within the system itself (if you did you would not have guarded against that risk!).

Encryption and data in the cloud

Most organisations have regarded data at rest within their on-premise information systems as being significantly less at risk than data on removable devices and data in transit, and therefore do not encrypt it.

But what about data in the cloud? There are certain features of cloud storage that may lead your organisation to wish to encrypt its data.  You may be concerned that a change of ownership or a change of management at your cloud provider might adversely impact on security.  You might be worried that the government of a territory in which the information is held may attempt to view the data.  You may be concerned about the employees of the cloud provider seeing the data.  However fundamentally speaking, data in the cloud is data-at-rest.  It is data that is being stored not communicated.  .  If you encrypt your cloud data you have the same problem as you would have if you encrypted data on your on-premise systems – how do you ensure that you maintain the encryption keys over time, and where do you store them?  If your reason for encrypting is a lack of trust of the cloud provider then storing the decryption keys with the cloud provider would defeat the object.

The challenge of encryption key management

Key management is crucial to any encryption model. In theory you want the organisation to give every individual a private/public encryption key pair. That means that not only can individuals encrypt information (with their public key) that they wish to keep secret.  But you also can provide a digital signature capability because they can encrypt with their private key a signature that anyone can read with that individual’s public key.  This offers proof that the individual alone must have signed it because it could only have been encrypted with the individual’s private key.  Thus the signature is in theory non-repudiatable (the indivdual could not deny that it was they that sent it).

There are a couple of interesting issues around key management.  Do you make it so that only the individual knows their private key? – in which case the digital signature is genuinely non-repudiatable.  But if the individual loses their private key then they lose access to their signature and to information that they have encrypted.  The alternative is that the organisation provides a way for the individual to recover their private key, for example through an administrator.  But this means that the digital signature is in theory now repudiatable because the administrator could have used it to sign a message.

I am currently reading a wonderful book that explains why even two decades into the digital age there is still no widely accepted digital signature capability – the vast majority of organisations do not use digital signatures (and hence still have a need to keep some paper records of traditional blue ink signatures) The book is called burdens of proof and is written by Jean-Francois Blanchette.

A unique feature of MoReq2010 when compared to previous electronic records management specifications is the provision for the DLM Forum to publish optional extension modules and plug-ins to extend the compulsory core modules of the specification.

This will enable the specification to embrace needs specific to some but not all organisations/sectors, without imposing costs on vendors who intend to develop products that do not service those organisations/sectors.

It will also enable the specification to develop over time to respond to new needs created by the ever-evolving world of applications used in business.

Yesterday afternoon at the DLM Forum members meeting in Copenhagen I heard Jon Garde announce a list of MoReq2010 extension modules and plug-in modules that would be developed over the coming 12 months.

None of these extension/plug-in  module will be compulsory.  Vendors will continue to be able to achieve MoReq2010 compliance with a product that does not meet any of them.  However vendors will be able to ask for their product to be tested against them.

Extension modules

Extension modules provide an extension to the core modules of the specification, but do not replace the core modules.  Below I have listed the extension modules that are due to be published in the next 12 months

Transformed records extension module

This  module will define a capability for a MoReq2010 compliant system to manage records that have been annotated and/or redacted.  The special issue around these relate to the fact that the system needs to manage the relationship between the unannotated/unredacted version of the document, and the annotated/redacted version(s).

File aggregations extension module

The core modules of Moreq2010 replaced the concept of a ‘file’ (which had been present in all previous ERM specifications)  with a much more flexible concept of an ‘aggregation’.  The concept of an aggregation refers simply to the containers that users use to organise their records. That could be anything from a folder structure to a SharePoint document library to an e-mail inbox.  The concept of an aggregation was made as flexible as possible in order to offer the vendors of products as diverse as e-mail clients, collaborative systems, wikis etc the possibility to secure MoReq2010 compliance.

However there may be some organisations who simply want a system that works like a traditional EDRMS, with a hierarchical classification, and with users restricted to creating files that can have sub-files and/or volumes and into which they can place documents.

The file aggregation module will define the capability for a system to hold aggregations called ‘files’ that can contain sub-files but cannot sprawl into multi-level folder structures.  This will also give backward compatibility to the predecessor MoReq2 specification.

Import services extension module

The import services module will define  the capability for a MoReq2010 compliant system to import data exported from any other MoReq2010 compliant system.  Of all the extension modules Jon mentioned, this is the most important to my mind.

The whole point of MoReq2010 is the idea that most records need to be migrated from one system to another at some point in their lifecycle, and hence every MoReq2010 compliant system must be able to export its records in the format specified by the export services module of MoReq2010, which is a core and compulsory module.

I predict that MoReq2010 will come to life if and when a vendor brings to market a product that complies with the optional import services extension module that is in the course of development.  Any organisation that deploys such a system as a records repository has a huge incentive to make its other applications MoReq2010 compliant.  It would know that the minute it wanted to replace such an application it could export all the content and import it into its records repository without a complex custom migration process.

Security categorisation services extension module

This will define a capability for a MoReq2010 compliant records system to implement security classifications in MoReq2010 such as secret, top secret etc.

Physical management services extension module

This will define a capability for a MoReq2010 compliant records system to manage physical objects (such as hard copy records)

E-mail client integration extension module

This will define a capability for a MoReq2010 compliant records system to integrate with an e-mail client such as Microsoft Outlook in order to capture e-mail as records into the system.

Plug-in modules

Plug-in modules for e-mail and for Microsoft Office documents

The function of plug-in modules within the MoReq2010 specifications are to provide alternative ways of implementing a particular type of functionality.

For instance a MoReq2010 system keeps content (for example electronic documents) in the form of components, that are managed as records.  The core modules of the specification contain a rather generic ‘electronic component’. Over the next 12 months two new plug-in modules will be written: one for e-mail and one for Microsoft Office documents.  These plug in modules will define the requirements necessary to capture and manage metadata specific to e-mail and Microsoft Office documents.

These formats have been chosen simply because they are in wide usage.  It is still possible to bring other formats into a MoReq2010 compliant systems – they will can  be brought in using the more generic electronic component.  More plug-in modules will be written in future years.

Jon anticipates that there will be a new version of MoReq2010 published annually to include all the new extension and plug-in modules

Between 2004 and 2006 Ben Plouviez (@benplouviez) oversaw the roll out of an EDRM (electronic documents and records management) system across what was then the Scottish Executive (but is now the Scottish Government).

Six years later and the system contains 14 million documents and is used by around 4,000 staff.

In this podcast Ben reflects even-handedly on both the benefits that having an organisation wide records repository has brought to the Scottish Government, and on the promises that the system has not fulfilled.

The roll out of the EDRM was driven partly by the Scottish Executive’s desire to breakdown silos between the various different parts of the administration. They  made the decision that wherever possible files would be open and accessible to the whole of the Scottish Government. There have been times when colleagues have found documentation that they would never have known existed were it not for the EDRMS.

The EDRM’s Scottish Government wide business classification scheme has not been an unqualified success, but nor could it be called a failure.  It is not terribly popular with users, who rarely use it to navigate to material that they wish to find.  However on the plus side the scheme has provided a stable and enduring  structure for the system.

Ben has found that the electronic files on the EDRM system do not tell a narrative in anything like as clear or as useable way as a typical paper file used to do.  Ben questioned whether it was feasible  for records managers to expect their organisations to keep a full electronic file of every piece of work they carry out.  Ben said that the concept of the file is predicated on the concept of the document and we are now seeing alternatives to the document in the form of blogs, wikis, discussion forums, etc.  None of these new formats fit naturally into the file.  I found it significant that MoReq2010 specification used the word ‘aggregation’ instead of the word ‘file’.  This implies that in the electronic world there are many different ways in which business communications can be collected (e-mails in in-boxes, tweets in tweet streams, etc..).

There have been some unexpected benefits to having an organisation-wide records repository.  For example Scottish Government have taken information from the system’s audit logs about who has read what on the EDRM and translated it into rdf triples (the non-proprietary format that underpins linked data and the semantic web).  They have then provided an interface to enable colleagues to query this data to find out what their colleagues have read on the system. This enables the serendipitous finding of documents of curerent interest, and provides a more human way of browsing and interrogating the system than that provided by either the business classification or by the search facility.  The Scottish Government have also used the same technique in relation to e-mail logs.  They have taken the records of who sent an e-mail to who and when, converted it to rdf, and provided a query and visualisation interface.  This means colleagues can find out who has been corresponding with particular colleagues or stakeholders.    Note that the content of the e-mail is not accessible, and that only e-mails with at least one person in cc have been included to ensure that private correspendence between two people is excluded.

Ben talked about the plans for the future of electronic records management in Scottish Government, including their intention to replace their existing EDRM within the next three or four years.  He speculated on whether it would be possible for one product/system to fulfill both their collabortion and records management needs, or whether Scottish Government would have to implement several different tools to deliver that vision.

This podcast is published as ECM Talk episode 014 – you can download it from here

This Thursday I went to the tea cloud camp meeting on cloud computing held at the National Audit Office in London for an update on progress with the UK Government’s G-Cloud

Launch of Cloud Store

We heard that the UK Government’s CloudStore  could go online as early as this weekend.

CloudStore will be in effect a catalogue of suppliers who have been accredited by the UK government to provide the government with cloud services.  The accredited companies are grouped under four headings – Infrastructure as a service, Platform as a service, Software as a service (including applications such as EDRM, CRM and collaboration) and Services (including systems integrators).

This is a technology shift towards cloud solutions,   but more importantly this is a procurement revolution. It is a move towards transparent pricing, with suppliers stating their prices up front on the CloudStore, and pay as you go, easy- to enter and easy-to-leave contracts.

One IT manager told us that in her career as a civil servant she had managed so many contracts with poor suppliers (she used a stronger term than poor!).  Even though the contractors were not performing her department had no choice but to keep them because they had no plan B.  The penalty clauses for leaving the contract early were so great as to make it uneconomical to change, and the length of the procurement process meant that their were no alternatives lined up ready and waiting to step in and fill the gap left by the ousted supplier. For her CloudStore means always having a plan B.   If she has a poor supplier in future she looks at the cloud store, finds an alternative, terminates the contract with the poor supplier and starts one with an alternative provider.

She identified one of the key thing about CloudStore was that we will increasingly see IT applications bought as commodities rather than as bespoke solutions.

The benefits should work both ways. The public sector get a better price and the suppliers will benefit from the lower cost of winning business.  They will be able to strike a deal with new public sector customers much more quickly.  Another potential benefit for suppliers is that CloudStore will be viewable online by anyone.  I would not be surprised if people in other sectors and other countries looked at the UK Government’s cloud store to get an idea of what suppliers have been accredited by the UK Government, what services they offer and what prices they offer.  There is also the capability for public sector bodies to write Amazon style reviews of the service they have received.

One of the speakers mentioned how pleased she had been with the response from suppliers. Hundreds of applications were received when the CloudStore OJEU issued late last year, and companies that did not apply first time around will be given further opportunities in future to apply to get onto the store.

G-cloud pilot – a County Council puts its e-mail into the cloud

We heard from a county council who were putting their e-mail into the cloud, as a pilot G-Cloud project. They had received six bids – three from vendors offering public cloud services, three from private clouds.  They narrowed it down to three bids – Microsoft’s Office 365, Google Apps, and IBM (who offered Lotus notes from a private cloud).  Each bid provided the functionality they wanted so they went on price alone (which tells you that e-mail, calendaring and basic collaboration is now a commodity).  Google Apps won.

The Council picked an initial group of around 150 volunteers to trial Google Apps.  In order to avoid a self selecting sample of technology enthusiasts they asked volunteers to give a reason why they wanted to join trial, and picked people with a range of different motivations. The volunteers were not given face-to-face training, but were each set up on Yammer so that they could act as a support community for each other. They have only received four calls to the service desk since it started.

One of the first things they found was how quick it was to bring people onto the service. They bought some servers to use to migrate users from their existing system (Lotus notes e-mail hosted in-house) to Google Apps in the cloud.  The servers will not be needed as soon as the migrations have all taken place.  They had 15 users up and running on the service within a week of signing the deal.

They have resisted the temptation to bring the whole organisations over to Google Apps in one big bang. Running two systems alongside each other brings with it inconveniences around calendaring – some staff are using Lotus Notes calendars and some using Google Apps so it is difficult for them to share appointments etc.  Their initial volunteer group of 150 people had to be expanded to 250 simply because some of the volunteers had colleagues that they needed to be on the same calendaring system with.

The Council are going to look in the spring at integrating Google apps with their EDRMS so that it becomes easier for colleagues to save e-mails needed as records.  They may also start working with Google Sites at some point (which would bring the implementation into the filesharing /collaboration space).

G-cloud and security

The Council said one of the benefits of G-cloud for them was that they did not have to think through on their own and from scratch  the questions of security in the cloud and personal data in the cloud.   A lot of the thinking had been done centrally, on a public sector wide basis   (with the caveat that individual public sector bodies still have to assess the risks arising from their own information systems and make decisions appropriate to that level of risk).

CESG (the Government’s National Technical Authority for Information Assurance) is carrying out information assurance checks on every service that applies to join the CloudStore framework, as part of the accreditation process.

CESG has come up with a classification of business impact levels (here is the pdf)to enable public sector bodies to assess the impact of any particular type of information being compromised.    Business impact level 2 corresponds broadly to the government security classification of ‘protect’.  This is information that the government does not want to see in the public domain, but if it got in the wrong hands  the damage would be more inconvenient than disastrous.  Business impact level 3 corresponds broadly to the government security classification ‘Restricted’ – this is information where there could be serious consequences (to individuals, organisations, commercial interests or the nation as a whole) if the information got into the wrong hands.

For example personal data whose compromise is  unlikely to put an individual in danger is likely to be regarded as impact level 2, whereas personal data whose compromise could put an individual in danger is likely to be marked as Impact level 3 or above.  Impact level 2 covers vast swathes of government work.

Both Google Apps and Microsoft’s Office 365 have been accredited up to Impact level 2.  We were told that some of the vendors had started to show an interest in being able to offer a service accredited for impact level 3 information, but for at least the short term the CloudStore would not be catering for impact level 3 information.

One IT manager told us that the point of the cloud services is that it caters for the majority of government’s needs, not for all their needs. She said it may be that public  bodies simply made separate provision for restricted documentation and e-mail – even if it meant having separate booths dotted around the office with computers staff could use for  ‘restricted’ communications.

G-cloud, data protection, and the issue of storing data outside of the EU

One of the big concerns with cloud adoption has been the 8th data protection principle (present in the data protection legislation of every EU member state) which states that personal data should not be transferred outside the European Economic Area unless that country or territory ‘ensures an adequate level of protection for the rights and freedoms of data subjects in relation to the processing of personal data’.

There is also  a wider concern that where information is stored out of the country, and particularly when it is stored outside the EU, then it comes under a legal framework that the UK cannot control (for example many countries have legislation giving their governments powers of inspection, on security grounds, of information held in their territory).

The speakers at the meeting referred  to Cabinet Office Guidance on Government ICT offshoring. The guidance states that no information with a national security implication should be stored outside the country (whatever the impact level).

Personal data is a slightly different matter-  the Cabinet Office guidance does not forbid personal data being stored outside the EU, provided measures are in place to ensure that the contractor treats the data in an ‘adequate’ manner (‘adequate’ meaning compliant with EU data protection principles and practice), and provided the security in the system is appropriate to the impact level of the information.  The guidelines give three ways of ensuring that a contractor operating from overseas has an ‘adequate’  data protection regime – safe harbor, model clauses and binding corporate rules.

The safe harbor scheme was set up jointly by the EU and the US.  Individual US companies that sign up for the safe harbour scheme are considered ‘adequate’ by the EU and therefore the UK public sector is not contravening this principle by storing such data with these companies.   The safe harbor arrangement has been criticised by some commentators.  Chris Connolly said ‘The Safe Harbor is best described as an uneasy compromise between the comprehensive legislative approach adopted by European nations and the self–regulatory approach preferred by the US’.     However this article from The Register last month predicts that the safe harbour arrangement will survive the proposed forthcoming overhaul of EU data protection legislation.

The second of the measures is model contract clauses with companies to ensure that the company operates ‘adequate’ protections in relation to the data it stores under the contract.  The European Commission has drawn up some such clauses and the so has the UK Government.

Binding corporate rules are where the Government accepts that the internal policies of a company operating both within and outside the EU are strong enough to ensure that an ‘adequate’ data protection regime is operated across the whole company (and not just inside the EU).  The guidance states that such corporate rules are an alternative to model contract clauses and must be approved by a relevant data privacy supervisory authority ( the Information Commissioner in the UK, or an equivalent in another member state).

Migrating content from one application to another is a problem that even now, two decades into the digital age, we have no solution for.  Migrating content is often so labour intensive and complex as to be non cost effective.   Any content migration involves compromises and ommissions that result in a significant loss of quality of the metadata that is held about the content being migrated.
Solving the content migration problem is about to become more urgent with the growing popularity of the software-as-a-service (SaaS) variety of cloud computing.  In this model the provider not only provides the software application, they also host your content.  Imagine what would happens if your organisation decided it wanted to change from one SaaS provider to another.  It wants to change from Salesforce to a different SaaS CRM.  Or it wants to go from SharePoint online to Box or Huddle or another collaboration/filesharing offering (or vica-versa).   What do you do?  How do you migrate content from SharePoint Online to Box? They have little in common in terms of how they are architected and what entities they consist of. What is the Box equivalent of a SharePoint content type?
The vendor lock-in problem is very real.  If you can’t migrate the content you are left paying two sets of SaaS subscriptions and managing two SaaS contracts.  If you were leaving because of a breakdown of trust with your original SaaS providor then how happy would you be leaving your content locked up with them on their servers?

Content migration is a problem that affects all organisations, and which affects archivists as much as records managers

The difficulty organisations experience in migrating content from one application to another matters in many situations. It matters when an organisation wants to replace an application with a better application from a different supplier. It matters in a merger/acquisition scenario, when an organisation wants to move the acquired company onto the same applications that the rest of the group are using.
It matters to archivists, because any transfer of electronic records from an organisation to an archive is, to all intents and purposes, a content migration.  I heard the digital preservation consultant Philip Lord say at a conference that the big difference for archives of the electronic world over the paper world is that:

• in the paper world it was possible for an archive to set up a routine process for transferring hard-copy records that it would expect all bodies contributing records to the archive to adhere to
• in the electronic world everytime an archive wishes to accept a transfer of records from a new information system it needs to work out a bespoke process for importing that content and its metadata from that particular system into their electronic archive.

Different applications keep their metadata in profoundly different ways

Migration from one application to another is extremely time consuming because you are:

• mapping from one set of entity types to another. Entities are the types of objects the application can hold (users/groups/documents/records/files/libraries/sites/retention rules etc)
• mapping from one set of descriptive metadata fields to another
• mapping from one set of functions to another. Functions are the actions that users can be permitted to perform on entities in the system (for example: create an entity/amend it/rename it/move it/copy it/delete it/attach a retention rule to it/grant or deny access permissions on it)
• mapping from one set of roles to another. Roles are simply collections of functions, grouped together to make it easier to administrate them. For example in SharePoint the role of ‘member’ of a site collects together the functions a user needs to be able to access a site and view and download content, and to contribute new content to the site, but denies them the functions they would need to administer or change the site itself.

Let us imagine we want to migrate content from application A to application B. Application A has an export function and can export all of its content and metadata into an xml schema. That is good. We go to import the content and metadata into application B. This is where we hit problems.

Application B looks at the audit logs of application A. They contain a listing of events (actions performed by users on entities within the system, at a particular point in time). Each event listed gives you the identity of the user that performed the function, the name of the function they performed, the identity of the entity they performed it on and the date or time at which the event occurred. Application B won’t understand these event listings. It is unlikely to understand the identifiers application A uses to refer to the entities and users. It is unlikely to understand the functions performed because application A has a different set of functions to application B.

Application B looks at the access control lists of application A. Each entity in application A has an access control lists that tells you which users or groups can perform what role in relation to that entity. Application B does not understand those roles, nor does it understand the functions that the roles are made up of. Therefore system B cannot understand the access control lists.

The end result is that application B cannot understand the history of the entities it is importing from application A, and it cannot understand who should be able to access/contribute to/change them.  It is also going to find it difficult to import things like retention rules, descriptive metadata fields, controlled vocabularies.

Migration reduces the quality of metadata

The process of migration is ‘lossy’.  In the world of recorded music it is said that when you move music from one format to another (LP to tape, tape to mp3 etc.) you cannot gain quality, you can only lose it.  When you migrate content from one system to another you cannot gain information about that that content, you can only lose it.  There will be whole swathes of metadata in system A that it will not be cost effective for you to map to conterpart metadata in system B.  You end up migrating content without that metadata, and your knowledge about the content that you hold is poorer as a result.

The fact that content migration is so labour intensive and lossy means that many organsiations opt to leave content in the original application and start from scratch in the new application.  This is a nice easy option, but there are downsides.   It means that the organisation has to maintain the original application for as long as it needs to keep the content that is locked within it.  This means paying the resultant cost of licence fees and support arrangements.   It also means a break in the memory of the organisation.  Users of the new system wishing to look back over previous years will have to go to the old system to view the content.  That is OK for a short period, during which time most colleagues will remember the old system and how to use it.  But as time goes by a larger and larger percentage of colleagues will have no knowledge or memory of the older system and how to use it.

The organisation may mitigate the impact of that by connecting the search capability of system B to the repository of system A.  The results of this are hit and miss.  The search functionality of system B will have been calibrated to the architecture of system B, it will not be calibrated to the architecture of system A.  Yes, it will return results but it will not be able to rank them very well (and you are still having to maintain system A in order that system B can run the search on it).

What can electronic records management specifications do to improve this situation?

The problem of content migration is not specific to records systems, it is a universal problem that affects any organisation wishing to move content from any kind of application to another application.

But it is a problem that is central to the concerns of records managers and archivists, because as a profession(s) we are concerned with the ability to manage records over time, and difficulties in migrating content hamper our ability to manage content over time.  We know that applications have a shelf life – after a period of years a new application comes along that can do the same job better and/or cheaper, and therefore we want to move to the new tool.  The problem is that retention periods for business records are usually longer than the shelf life of applications.    Therefore it is probably from the records management or archives world that a solution will come to this problem, if it comes at all.

The first generation of electronic records management system specifications (everything from the US DoD 5015.2 that first came out in 1998 to MoReq2 which came out in 2008), did not attempt to tackle the problem. They told vendors what types of metadata to put into their products – but they did not tell vendors how to implement that metadata.   For example these specifications would specify that records had to have a unique system identifier, but it was up to the vendor what format that identifier took. They had to have a permissions model but what functions and roles they set up was up to the vendor, and so on.

This lack of prescription had the benefit of sparing vendors of existing products the necessity of re-architecting the way they assign identifiers/implement a permissions model/ keep event histories etc. Had existing vendors been forced to re-architect in such a way it would have proved a major disincentive for them to produce products that complied with the specification. But the disadvantage was that the electronic document and records management systems (EDRMS) that these specifications gave rise to each had their own permissions models and metadata structures. When an organisation wanted to change from one specification compliant EDRMS to another, they had the same content migration problems as you would when migrating content between instances of any other type of information system. An archive (for instance a national archive) wishing to accept records from different EDRM systems would need to come up with a bespoke migration procedure for each product.

MoReq2010’s attempt to facilitate content migration between MoReq2010 compliant systems

MoReq2010 marks something of a break with past electronic records management specifications.  One of its stated aims is to ensure that any compliant system can export its content together with their event history, their access control list and their contextual metadata, in a way that any system that has the capability of importing MoReq2010 content can understand and use.

In order to this it has had to be far more prescriptive than previous electronic records management specifications in terms of how products keep metadata.

For example

• It tells any compliant system to give each implementation of that system a unique identifier. This means that any entity created within that implementation will be able to carry with it to subsequent systems information about the system it originated in
• It tells every implementation of every compliant system to give each entity it creates the MoReq2010 identifier for that entity type, so that any subsequent  MoReq 2010 compliant system that the entity is migrated to understands what type of thing that entity is (is it a record? or an aggregation of records? or a classification class or a retention schedule? or a user? or a group? or a role?)
• It tells every implementation of every compliant system to give every entity created within it a globally unique identity an identifier in a MoReq2010 specified format. Each entity can carry this identifier with it to any subsequent MoReq 2010 compliant system, no matter how many times it is migrated
• It tells every implementation of every compliant system to give each entity an event history that not only records the functions performed on that entity whilst it is in the system, but which also could be carried on and added by each subsequent system.
• It tells each compliant system to create an access control list for each entity in the system, that governs who can do what in relation to that entity whilst it is in the system, and which can be understood, used, and added to by any subsequent compliant system that the entity is migrated to.

To achieve the last two of these ambitions MoReq2010 had to get into the nitty gritty of how a system implements its permissions model.

MoReq2010 and permissions models

I recorded two podcasts with Jon Garde about the permissions model in MoReq2010:

• episode 7 of Musing Over MoReq2010 is about how the ‘user and group service’ section of the MoReq2010 specification
• episode 8 (shortly to be published here )is about the ‘model role service’ section – the part of the MoReq2010 specification that deals with functions (the actions users can perform within the system) and roles (collections of functions).

In the latter podcast Jon said that the model role service was the part of MoReq2010 that caused him the most sleepless nights when he wrote it.  The problem was that every product on the market already has a permissions model, with its own way of describing the functions that it allows its users to perform on entities within the system.

The dilemma for Jon writing Moreq2010 was as follows:

• If the specification prescribed a way for each system to implement its permissions model then existing systems would have to be rewritten and this would act as a major disincentive for vendors to revise their products to comply with MoReq2010
• If the specification did not prescribe a way for each system to describe the functions that users could perform within it then subsequent systems would not be able to understand the event histories of exported entities (because it would not understand which actions had been performed on the entity concerned) or their access control lists (because it would not understand what particular users/groups of users were entitled to do to that entity)

The solution that Jon adopted was half way between these two options.  In the model role service MoReq2010 outlines its own permissions model, with definitions of a complete set of functions that a record system can allow users to perform on entities.

MoReq2010 does not insist that to be compliant a system must implement every one (or even any one) of the functions that are outlined within the model role service.  It allows products to carry on using their own permissions model.  However MoReq2010 does insist that a system must be able to export their content and metadata with the functions and roles expressed as the functions and roles outlined in the MoReq2010 specification.  In other words a product would need to map its existing permissions model (functions and roles) to MoReq2010 functions and roles.   This would mean that two MoReq compliant systems with entirely different permissions models could both export their content with all of the functions in the access control lists and the event histories expressed as MoReq2010 functions.

Mapping the functions and roles in their product’s permission model to MoReq2010’s permission model is a significant body of work for vendors of existing systems, and they will obviously make a commercial judgement as to whether the benefit to them of achieving MoReq2010 compliance outweighs the cost of the investment they will need to make those mappings and to implement the other changes, such as the identifier formats, that MoReq2010 demands.

Because MoReq2010 is so prescriptive as to how systems keep metadata it could well be that it is easier for new entrants to the market to write new products from scratch to comply with the specification than it is for existing vendors to re-architect their products to comply. If I was a vendor writing a new document or records management system from scratch I would certainly think about simply implementing the MoReq2010 permissions model outlined in the model role service.

Why is import more complex than export?

The core modules of MoReq2010 include an export module.  Every compliant system must be able to export entities and their event histories, access control lists and contextual metadata in a MoReq2010 compliant way.   There is no import module in the core modules of MoReq2010.  Vendors can win MoReq2010 compliance for their products without their products being able to import content and its metadata from other MoReq2010 compliant systems.

The import module of MoReq2010 is being written as I write, and is scheduled for release sometime in 2012.  It will not be compulsory.  The reason why the import module is not a compulsory module of the specification is that not all records systems will need to import from other MoReq2010 compliant records systems.  For example by definition the first generation of compliant systems will not have to import from other compliant systems (because they have no predecessor compliant systems to import from!).

It will be more complex for a system to comply with the import requirements of MoReq2010 (when the module is published) than it is with the export requirements.

For example:

• an existing product that seeks compliance with the core modules of MoReq2010 (but not the additional and optional import module) will have to map its functions (actions/permissions) and roles to the functions and roles outlined in MoReq2010.  It does not have to worry about all the functions listed in MoReq2010 – only the functions that it needs to map its own functions to
• a product that seeks additionally to comply with the import module of  MoReq2010 compliant system will need to be able to implement all of the functions listed in MoReq2010 – because it needs to be able to import content from any MoReq2010 compliant system and a MoReq2010 compliant system may chose to use any of the functions listed in MoReq2010.

I put it to Jon in our podcast on the model role service that we would know that MoReq2010 had ‘arrived’ if and when someone brings to market a product that complies with the import module and is capable of importing content from MoReq2010 compliant systems.  Once you have products capable of importing from MoReq2010 compliant systems there is all of a sudden a purpose to implementing MoReq2010 compliant systems – the theoretical possibility of being able to pass content onto another system that understands the content as well or nearly as well as the originating system is turned into a practical reality.  Once you have a product that is capable of importing from MoReq2010 compliant systems it is in the interests of anyone implementing that product to influence whoever runs the applications that they wish to import from to make those applications MoReq2010 compliant. Imagine a national archives running an electronic archive with a MoReq2010 import capability.  It would be in the interests of that national archives to pursuade the various parts of government who contribute records to them to implement MoReq2010 compliant systems.

Jon’s response on the podcast was to lay down a challenge to the archives world to develop a MoReq2010 compliant electronic archive system, with a MoReq2010 compliant import capability.

What are the chances of MoReq2010 catching on?

MoReq2010 is doubly ambitious.  In this post I have looked at its ambition to ensure that content can take its identifiers, event history, access control list and contextual metadata with it through its life as it migrates from one system to another.  Its other great ambition is to reach a situation where any application in use in a business is routinely expected to have record keeping functionality.   The two ambitions are related to each other.

• MoReq2010 makes it feasible for the vendor of a line of business system to add records management functionality to their product and get it certified as being a compliant records system. The specification has done this by eliminating from the core modules  any requirements that are would not be necessary for every system to perform however small and however specialised. A compliant system does not have to be able to do all the things an organisation-wide electronic records management system would have to do.  It only needs to be able to manage and export its own records. Note that MoReq2010 makes it possible for vendors of line of business systems to seek compliance, but the specification alone cannot incentivise them to do this – incentivisation would have to come from the market or from organisations that could influence the market
• Because MoReq2010 allows the possibility for  records to be kept in multiple line of business and other systems within an organisation then the issue of migration becomes very important.  When a line of business applicatin is replaced the organisation will need to migrate content  either to the application’s replacement or to an organisational records repository or or to a third party archive. Hence the ambition that any compliant records system can export content and metadata in a way that another compliant system can understand.

Being ambitious carries with it a risk.  MoReq2010 does call for existing vendors to re-architect its systems, and vendors do not like re-architecting their systems.  If too few vendors produce products that comply with the specification then MoReq2010 will go the way of its predecessor, MoReq2, which died because only one vendor felt it was commercially worthwhile to produce a product that complied with it.

In the situation that electronic records management finds itself in, being ambitious is less risky than trying to incrementally tweak previous specifications.   MoReq2, failed because by the time it was published in 2008 the bottom had fallen out of the market for the EDRM systems that it and previous electronic records management system specifications underpinned.  SharePoint had come along and pushed it over like a house of cards.

EDRM fell without so much as a whimper because no-one was prepared to defend it.  Archivists were not prepared to defend it because they had not benefited from it – it was as hard for them to accept electronic transfers from EDRM systems as from any other type of application.  Practioners were not prepared to defend it because it had proved difficult and expensive to implement monolithic EDRM systems across whole enterprises.  The ECM vendors who had acquired EDRM products were not prepared to defend it because EDRM represented only a relatively small portion of their portfolio, and they had no stomach for a fight with Microsoft.

MoReq2010 has a chance of success.  It is not guaranteed to succeed, but it has a chance.  The reason why it has a chance is because it is addressing the right two questions – how do we get records management functionality adopted by all business applications? and how do we ensure that content can be migrated easily and without significant loss of metadata from one application to another?

These questions will have to be nailed. If MoReq2010 succeeds in nailing them so much the better.  If it doesn’t, if the market isn’t ready for it, then whatever specifications come after it will have to nail them.  There is no going back to the EDRM ‘one records system-per-organisation’ model.