Question time at the Records Management Society Conference

Last Monday I had the pleasure of sitting on a panel at the RMS conference to answer a set of questions on the current state of electronic records management.  

Paul Duller chaired and the panel consisted of Jon Garde, Marc Fresco, Leanne Bridges and myself .

We were lucky enough to have had a very relevant and challenging set of six questions submitted to us in advance via the RMS conference website.

What follows are my recollections of some of the points that we made.   

1. Is Moreq2 a useful standard for supporting electronic information management systems or has it come too late to make a difference?

Jon:   MoReq 2 took a long time to come to fruition, and that was frustrating.  It is not a significant advance upon previous standards (TNA 2002) and is if anything longer and more complex for vendors.   But there is great potential in MoReq 2.   MoReq2 isn’t finished with the publication of the specification.  It is an ongoing project, and once the governance framework and testing regimes are in place we will start to see value

Leanne:  At the Audit Commission we used MoReq1 in our procurement of an EDRMS system,  these standards are useful in giving organisations confidence they are procuring a system that can do the job.  

Me:   No electronic records management standard published in 2008 could hope to have the same influence as the first standard published by the Public Record Office (now the National Archives) in 1999.  Back then vendors were unsure of how to do electronic records management, and the standard was able to create the market for EDRMS.  Now the market has matured.  There is much broader understanding of electronic records management among vendors and practictioners alike.    EDRM has morphed into ECM,  which adds into the mix functions that are out of the scope of records management standards, such as collaboration,  web content management and basic social networking functionality.  And a big gorilla has entered the market in Microsoft with its SharePoint product.  Microsoft has been able to sell SharePoint to organisations looking for an ECM without baseing its records management model on standards like MoReq2.

Marc:   Its a fact that many EDRMS implementations have failed.  But it is also a fact that many other projects has succeeded.  The success or failure is ascribable at least as much to the project management and change management efforts around the technology, not the technology itself.  MoReq 2’s relevance is shown by the number of languages it has been translated into.

2. There is a general belief out with the RM community that MS SharePoint implementation is relatively simple and easy and that EDRMS implementation is hard and burdensome on users and their organisation. What evidence can the panelists provide from their experience that supports or is at odds with this belief?

Me:   I am not sure that ‘easy’ is the right word for SharePoint:  the simple old shared drive is much easier than SharePoint!   For teams the advantage they get from SharePoint is that they it gives them more choice on how they set up their information environment.  But choice brings with it some elements of complexity.     Within their SharePoint team sites teams are having to decide how many document libraries they want,  what they want them for, whether they want version control enabled, whether they want to organise their libraries by folders or content types.  When they start a new piece of work they need to decide whether they want a whole new sub site, or a new document library, or a new folder within a document library. 

For an organisation SharePoint is certainly far from easy to govern:  it takes considerable ongoing thought and effort to control the creation of new sites to prevent SharePoint sprawl, and the SharePoint records centre is far from straightforward to implement if you wish to use that for records management.

Marc:  SharePoint is easy and cheap to implement so long as you don’t want to implement it well.  If you want to implement it well  then you will have to give just as much thought to issues of how you want SharePoint to be structured, how you set up metadata and search, and  about governance and access permissions, as you would with an EDRMS implementation.  And if you want to manage records well all commentators are agreed you are going to have to plug in an EDRMS capability at the back of it anyway.


3. Is there a future for independent EDRM vendors now that Microsoft is targeting this market?


Me:  the next version of SharePoint will be out in the first half of next year and will be called SharePoint 2010.  Although no one has seen it yet, industry analysts are predicting that there won’t be significant improvements in the records management capabilities.  [see Kathleen Reidy and CMS Wire].  I agree with the analysts:  adding MoReq 2 records management functionality to SharePoint would be a major rewrite of the software and I can’t see Microsoft  undertaking it in the near future.    

Microsoft are under no competitive pressure to improve records management,   they have been able to sell perfectly well without it, and none of their real competitors (Google, Apple, Amazon cloud computing) are offering records management capabilities.    The EDRMS vendors recognise that they are not big enough to compete with Microsoft and are falling over  backwards to develop offerings that plug into the back of SharePoint rather than trying to take it on.  

Cloud computing is a threat to Microsoft because Amazon and Google (with their massive data centres) have got into that space first.  Microsoft are moving into the cloud computing market now, and have developed Azure:  an operating system similar to Windows that Microsoft would host from their datacentres rather than organisations having to install it on their servers.  But this means that much of the development effort in upgrading SharePoint in future will go into making sure it can run in the cloud on Azure as well as on Windows.

Jon:  Microsoft have realised that the recent  ISO standards for document formats (Open Document Format and Office XML) threatens the monopoly they had through Microsoft Office.  The move to standards based formats means that documents produced in MS Office are now in an XML format and hence competitors  can offer word processing packages (such as Google docs and Open Office) that can produce documents to the same format.

SharePoint enables Microsoft to keep Office going, because they have tightly integrated SharePoint with Office, and provided functionality to Microsoft Word, Excel and Powerpoint users that aren’t available to users of  the equivalent products of their competitors.  In fact I woudn’t be surprised to see in future years Microsoft binding SharePoint in with Windows, so that any organisation that bought Windows got SharePoint embedded in with it.

There will always be a role for the independent providers,  that is where innovation comes from.


4. In light of the Government White Paper on Open Source and Open Standards, how can organisations justify not using an Open Source solution for RM challenges?

Marc:  It is going to be very interesting to see the progress of open source systems over the next few years.   In EDRMS sphere the open source offerings are relatively new to the market compared to the proprietary systems,  but  Alfresco are an open source company that has advanced plans to go for MoReq2 compliance.

Jon:  As a programmer I love the idea of open source.  But if I was an organisation going open source I wouldn’t start with EDRMS.  I would start by looking at the whole stack that the organisation was using:  for example an organisation with a standard Microsoft stack (Microsoft Office running on Windows servers, with SQL as the main database) could think about whether it wanted to run  open source alternatives (such as Open Office, Linux servers,  MySQL database).   And then think about whether it wanted to go for an open source EDRMS to manage the information on that stack.  Doing it this way round also gives time for the  open source EDRMS offerings to develop and mature.

One motivation for the Government’s paper on open source could be a gambit to put price pressure on Microsoft:  it is the UK government saying to Microsoft ‘don’t raise your prices to much because we are willing to move to open source alternatives’.

Leanne:  Open source is great but it should be done on a pragmatic basis:  I know someone whose organisation is trying to go open source for all their information systems:  and is finding it difficult in some areas to find good open source solutions, particularly where the solution needed is very specialised.

Me:  It found it interesting listening to Jeremy Tuck speak at a conference about Islington Council’s implementation of the open source ECM system  Alfresco.   In order to consider the Open Source solution the Council to adapt their procurement  procedures.  The proprietary systems had salespeople who prepared costed bids.  Islington had to prepare a statement of what the costs and benefits would be of taking the Open Source code for Alfresco and implementing it as an ECM, and then compare this with the bids from the proprietary vendors.  


 5. What do the panelists see as the future role for the Records Manager in the world of electronic information management?

Leanne:  a consultant and advisor to the rest of the organisation

Jon:  a mentor to people in the organisation

Me:  If  services like Facebook and Twitter continue to grow at their current rate  then in a year or two’s time almost everyone of working age will be on one of those services.   Blocking access to these services is easy for organisations when staff are using devices their employer has bought for them, on the organisation’s premises.   But it is not so easy  when people bring their own computing devices to work (netbooks cost less than £200. iPhones and other smartphonest are spreading like wildfire) nor when staff work from home.  And when customers start to want to contact people in the organisation by Twitter or Facebook it is a brave organisation that will cut themselves off from their customers.  Will working conversations start to migrate away from e-mail on to systems like Twitter and Facebook? If so what do organisations do?  Do they rely on staff copying important communications into organisational systems?  Do they deploy systems to interoperate with these Web 2.0 services?

Evaluation of records management in SharePoint 2007

Records management is the weakest link in SharePoint 2007. This weakness will not deter organisations from adopting the product. SharePoint’s strengths in other areas, particularly the flexibility it provides teams in setting up their own collaborative environment, together with the strength of Microsoft’s marketing position, has ensured the product has seen phenomenal uptake since its launch in November 2006.

Microsoft have come up with an innovative records management model of their own for SharePoint. They have not attempted to follow the EDRMS model laid out in MoReq 2 or TNA 2002, where a system must have the capability to hold a corporate records classification (fileplan), which can be linked to a set of retention rules and a set of access rules. The EDRMS model gives strong corporate control as these system require teams creating new folders for new pieces of work to place them within the fileplan where they inherit the appropriate retention and access rules. The cost of that control however is the need for teams to interact with a large corporate hierarchical classification.

The SharePoint records management model involves giving teams document management capabilities within document libraries in team sites,  whilst providing a separate record centre facility for the protection of documents needed as records, and for the application of retention rules to them. This model has the advantage of avoiding imposing a corporate classification on teams, allowing teams to organise their team sites the way they wish.  However in practice implementation of the records centre has not been easy for those organisations that have tried it. They have found it difficult to find robust processes for selecting and routing documents needed as records from the team site to the right place within the records centre.

The way SharePoint records management works is closely tied in with the way that document works in the product. In this post we will look first at how document management works in SharePoint 2007, then at how retention rules work, then  at some of the specific challenges of setting up and administering the SharePoint records centre.


How document management works within SharePoint

In SharePoint teams collaborate on documents within document libraries, which exist within team sites.

Version control

When setting up a document library teams have the choice of whether or not to enable version control and check in/check out.

Metadata (SharePoint content types)

If a team wants specific metadata to be captured about documents within a document library they do so by setting up one or more SharePoint content types for that document library. The team can either create entirely new content types, or they can choose from content types that the organisation has already set up within their SharePoint site collection.

When a team sets up a content type they determine which columns (fields) of metadata should be captured about those documents. They can set up pick-lists for any column that requires a controlled vocabulary. However out of the box SharePoint only supports flat pick-lists, not hierarchical controlled vocabularies.

SharePoint does not automatically allocate a unique system identifier (number or code) to documents saved into the system. This is a weakness as a unique identifier is a simple but very precise and useful piece of metadata.

If a team wants to set up document templates,  then they can do so by linking the template to a content type.  This enables a colleague to create a new document from within a SharePoint document library.

Organising document libraries: folder structures versus content types 

A team can use a folder structure to organise a document library, in which case it functions much like a shared drive. Alternatively they can set up SharePoint content types to allow colleagues to sort or filter the library by the different columns of metadata for each document: much as you would an Excel spreadsheet.

SharePoint 2007 is written to favour content types rather than folders for organising document libraries.

Folders in SharePoint have no more functionality attached to them than folders in a shared drive. For example you can not link a retention rule to a folder in a SharePoint document library. 

Content types are pivotal to SharePoint 2007. Without content types each document library is a silo. Content types offer the possibility to break out of silos. Where an organisation has many different teams doing a similar type of work, they can define a content type for that type of work. If each different team sets up that content type within the document libraries that they use for that work, then the content type provides a bridge bringing all that work together.

Retention rules can be linked to content types. However the content type will not apply that retention rule to documents within the team site. The content type will only apply the retention rule to a document once the document has been copied across to a ‘records centre’ within SharePoint.

Managing content types

Content types poses change management challenges, governance challenges and practical challenges:

  • For teams coming new to SharePoint from shared drives, content types are the main difference between a shared drive and a document library.  Some teams are enthusiastic about the possibilities they offer, but others are wary and prefer to have a straightforward folder structure which has the advantage of familiarity. Helping teams make an informed choice between the two options is an area records managers can add considerable value within SharePoint implementations.
  • It is very hard to stop content types proliferating. Out of the box SharePoint content types can be set up locally by a team site owner, or defined across an entire site collection. The risk is that teams set up a document library, are unaware that an appropriate content type already exists, and therefore create a new one. To get the most out of SharePoint content types an organisation would think through what metadata columns and content types it needed at the very start of the implementation. The organisation would also put governance arrangements in place so that when teams need a content type for their document libraries they would be required to either choose an existing one or justify the need for setting up a new one.
  • SharePoint is asking content types to do several different things: to determine what metadata is captured about documents, to link retention rules to documents, and to specify a document template for documents. Sometimes there is a tension between these three roles. 
  • A content type only functions within the boundary of a site collection: if you have more than one site collection you will need to set up new content type for each site collection. (Read this post for an excellent description of the SharePoint site collection and the reasons why you may need more than one of them)


Alternative ways to apply retention rules within SharePoint 2007

Retention rules in SharePoint are called expiration policies. SharePoint can apply expiration policies to libraries: document libraries in team sites, or records libraries in the Records Centre.
You have two options. You can either:

  • Link your expiration policy to document libraries within team sites OR
  • Link your expiration policies to content types and route documents needed as records to the SharePoint records centre where they will inherit the retention rule of the content type

Linking expiration policies to document libraries

Linking your expiration policies to document libraries within SharePoint has the advantage that the policy is applied to documents in situ: you do not have to move them to a records centre.  However it does constrain teams to restrict document libraries to records sharing a common retention rule, as you can only apply one retention rule per document library.  Microsoft never intended document libraries to be used for storing records: they intended that documents needed as records would be sent to the records centre.  The expiration policy on the document library is merely to ensure that  documents not needed as records are disposed of promptly.  Document libraries do not protect documents against amendment or deletion.  

I have not yet heard of an organisation that has taken the route of attempting to store its records in team site document libraries and apply expiration (retention) policies to those document libraries.   The significant administrative overhead in implementing the records centre option may tempt some organisations to use this route.

Linking expiration policies to content types

Microsoft’s intended model for records management within SharePoint involves the use of content types to link retention rules to records and to route records to a records centre.  In the records centre documents are protected from unauthorised amendment and deletion and the retention rule is applied to them.  

The big disadvantage of applying expiration policies via content types is that SharePoint will only apply that expiration policy to those documents once they have been routed from documents libraries to the SharePoint records centre. This gives you the significant administrative overhead of ensuring that documents from that content type are moved to the right place in the records centre at the right time. 


Critical assessment of the SharePoint records centre

This is how Microsoft intended records management to work within SharePoint 2007 (out of the box, no customisation)

  • Expiration policies (retention rules) are defined
  • Content types are defined, and have expiration policies linked to them.
  • Document libraries are set up within team sites, with content types enabled
  • A records centre is set up within SharePoint
  • A records routing table is set up for each content type (this tells SharePoint where to locate documents of that content type when they are routed to the records centre)
  • Colleagues collaborate on documents within document libraries in team sites
  • When colleagues wish to save a document as a record they right click on it, and choose the option ‘send to records centre’.
  • When the option ‘send to records centre’ is chosen, SharePoint takes a copy of the document and sends the copy to the records centre. It routes the document to the records library within the records centre designated for that content type. In the records centre the document is protected from amendment or deletion for as long as the retention period specified for that content type

The fact that SharePoint forces us to move documents from the team site to the records centre in order to protect them and apply retention rules to them poses us the following major challenges:

  • How do we ensure that all or most documents needed as records are routed to the records centre?
  • How do we avoid losing all the context that the document had when it sat within the team site?
  • How do we ensure that the document is routed to the right place in the records centre?
  • What do we do when teams use methods other than documents to record their work?
  • How do we ensure that the records centre becomes a useful resource, that repays the resources expended in setting it up and maintaining it?

Ensuring all or most documents needed as records are routed to the records centre

Routing documents to the records centre depends upon colleagues right clicking on the document as it sits in their team site document library, and selecting the option ‘send to records centre’. The likelihood is that many colleagues will omit to take this step. Sending a copy of the document to the records centre adds no appreciable value to the team that created the document, who already have access to it within their team site.

DEFRA have tried an innovative approach of asking colleagues to indicate when they save a document whether a document is is of high, medium or low importance when they save it within a team site document library.   DEFRA have customised SharePoint to automatically route across to the record centre any document defined as high or medium importance that has not been changed for six months.

Avoiding loss of context when the document moves from the team site to the records centre

When a document is saved to a document library within a team site it possesses some context. That context is provided by the other activity that is taking place within the team site: the discussion forum, calendar, announcements, wikis, blogs, any other document libraries etc. It is also provided by the name of the team site itself, and by the navigation pathway to the team site. When a document is routed away from the team site to the records centre it loses most of that context.

Out of the box when a document is routed from a team site to the records centre it carries with it the metadata about the document itself, and the name of the document library that it sits in.    DEFRA customised the routing of documents to the records centre in order to ensure that a document additionally brings with it details of the navigation pathway down to the team site.

Ensuring that documents are routed to the right place within the records centre

Microsoft envisages that for every content type you create you have a designated records centre library ready to receive records of that content type. When a document is sent to the SharePoint records centre it is routed to whichever records centre library that content type has been linked to in the records routing table. One records centre library may be set up to receive records of more than one content type.

If a colleague sends a document to the records centre that is not allocated to a content type, or which is allocated to a content type that does not have a routing table set up for it, then the document is simply sent to an ‘unclassified’ library within the records centre.

Challenges arise because content types are best suited for document types (invoices, reports, contracts etc), whereas  the record of any particular piece of work (a project, a programme etc.) usually contains a variety of different document types.

Don Lueders has written this useful blog post describing how a client of his customised SharePoint so that they could set up record centre libraries for different types of work (rather than different types of document). The work around involves getting the records centre to determine which library to send the document to based on a metadata field within the content type rather than the content type itself: the metadata field in question can be set to capture the name of the piece of work.

Capturing records when teams use methods other than documents to record their work

Within a SharePoint team site, documents are just one of the means a team can use to record their work. They may alternatively choose to record an aspect of a piece of work with a wiki, a blog, a discussion forum. Information recorded within these outputs cannot be routed to a records centre. The records centre is only geared up to accept documents.

The most serious example of this relates to the ‘Document workspace’. The document workspace is a facility that allows a team to create a collaboration area around one particular document. This is useful where a document requires the input of several people, and it allows colleagues to have a discussion around that document within the workspace, as well as manage different versions of the document. SharePoint allows you to save the actual document produced in the document workspace to a document library and then route it to the records centre. SharePoint does not enable you to roll up the discussions that took place within the document workspace and send them to the records centre with the document.

Ensuring that the records centre becomes a useful resource, that repays the resources expended in setting it up and maintaining it

Records exist to satisfy the need of different stakeholders to understand a particular work. When we think back 20 years the hard copy file in a traditional filing system was trusted because it was the tool that the people who carried out the work used whilst they were working. In SharePoint the working tool is the team site, and my prediction is that most stakeholders in a piece of work are likely to get a better impression of that work by looking at the team site, than by looking at the selection of documents siphoned off from the team site to go to the records centre. I do not anticipate heavy usage of the records centre as a reference tool. Colleagues will have gained familiarity with team sites from working in their own team site, but will have little or no familiarity with the records centre. If the SharePoint records centre is not used as a reference tool, then it will not justify the considerable administrative effort necessary to sustain it.



It is important for the future of records management that we come up with an electronic records management model that enables us to capture records from a team without shackling them with a corporate file plan. I applaud Microsoft’s attempts to come up with such a model, but I don’t think that the SharePoint records centre is there yet. As it stands at the moment it is too fiddly, requires too much customisation and too much administrative effort to sustain, and yields too little information retrieval benefit.

It is hard enough governing team sites and document libraries within SharePoint. The records centre option means that we are additionally having to ensure that a records centre library is set up for each area of work, that the appropriate content type is set up and enabled for each document library in each team site, and that the routing table for each content type sends documents from the document library to the appropriate records centre library.

My prediction is that the majority of organisations that deploy SharePoint 2007 will not deploy the records centre. Those organisations that acknowledge a serious records management need will, if funds allow, use a third party system to apply classification and retention rules to records. Those organisations that do not do not acknowledge a serious records management need will concentrate their efforts on governing SharePoint to prevent SharePoint sprawl, and will live without the ability to apply retention rules (much as organisations have limped on with shared drives despite its lack of records management functionality).

Note:  most of the links from this post point to Don Lueders’ excellent SharePoint records management blog:  if you are intending to deploy the SharePoint records centre it is well worth reading through the archives of Don’s blog for explanations of how it works and tips on customisations.