Evaluation of records management in SharePoint 2007

Records management is the weakest link in SharePoint 2007. This weakness will not deter organisations from adopting the product. SharePoint’s strengths in other areas, particularly the flexibility it provides teams in setting up their own collaborative environment, together with the strength of Microsoft’s marketing position, has ensured the product has seen phenomenal uptake since its launch in November 2006.

Microsoft have come up with an innovative records management model of their own for SharePoint. They have not attempted to follow the EDRMS model laid out in MoReq 2 or TNA 2002, where a system must have the capability to hold a corporate records classification (fileplan), which can be linked to a set of retention rules and a set of access rules. The EDRMS model gives strong corporate control as these system require teams creating new folders for new pieces of work to place them within the fileplan where they inherit the appropriate retention and access rules. The cost of that control however is the need for teams to interact with a large corporate hierarchical classification.

The SharePoint records management model involves giving teams document management capabilities within document libraries in team sites,  whilst providing a separate record centre facility for the protection of documents needed as records, and for the application of retention rules to them. This model has the advantage of avoiding imposing a corporate classification on teams, allowing teams to organise their team sites the way they wish.  However in practice implementation of the records centre has not been easy for those organisations that have tried it. They have found it difficult to find robust processes for selecting and routing documents needed as records from the team site to the right place within the records centre.

The way SharePoint records management works is closely tied in with the way that document works in the product. In this post we will look first at how document management works in SharePoint 2007, then at how retention rules work, then  at some of the specific challenges of setting up and administering the SharePoint records centre.

 

How document management works within SharePoint

In SharePoint teams collaborate on documents within document libraries, which exist within team sites.

Version control

When setting up a document library teams have the choice of whether or not to enable version control and check in/check out.

Metadata (SharePoint content types)

If a team wants specific metadata to be captured about documents within a document library they do so by setting up one or more SharePoint content types for that document library. The team can either create entirely new content types, or they can choose from content types that the organisation has already set up within their SharePoint site collection.

When a team sets up a content type they determine which columns (fields) of metadata should be captured about those documents. They can set up pick-lists for any column that requires a controlled vocabulary. However out of the box SharePoint only supports flat pick-lists, not hierarchical controlled vocabularies.

SharePoint does not automatically allocate a unique system identifier (number or code) to documents saved into the system. This is a weakness as a unique identifier is a simple but very precise and useful piece of metadata.

If a team wants to set up document templates,  then they can do so by linking the template to a content type.  This enables a colleague to create a new document from within a SharePoint document library.

Organising document libraries: folder structures versus content types 

A team can use a folder structure to organise a document library, in which case it functions much like a shared drive. Alternatively they can set up SharePoint content types to allow colleagues to sort or filter the library by the different columns of metadata for each document: much as you would an Excel spreadsheet.

SharePoint 2007 is written to favour content types rather than folders for organising document libraries.

Folders in SharePoint have no more functionality attached to them than folders in a shared drive. For example you can not link a retention rule to a folder in a SharePoint document library. 

Content types are pivotal to SharePoint 2007. Without content types each document library is a silo. Content types offer the possibility to break out of silos. Where an organisation has many different teams doing a similar type of work, they can define a content type for that type of work. If each different team sets up that content type within the document libraries that they use for that work, then the content type provides a bridge bringing all that work together.

Retention rules can be linked to content types. However the content type will not apply that retention rule to documents within the team site. The content type will only apply the retention rule to a document once the document has been copied across to a ‘records centre’ within SharePoint.

Managing content types

Content types poses change management challenges, governance challenges and practical challenges:

  • For teams coming new to SharePoint from shared drives, content types are the main difference between a shared drive and a document library.  Some teams are enthusiastic about the possibilities they offer, but others are wary and prefer to have a straightforward folder structure which has the advantage of familiarity. Helping teams make an informed choice between the two options is an area records managers can add considerable value within SharePoint implementations.
  • It is very hard to stop content types proliferating. Out of the box SharePoint content types can be set up locally by a team site owner, or defined across an entire site collection. The risk is that teams set up a document library, are unaware that an appropriate content type already exists, and therefore create a new one. To get the most out of SharePoint content types an organisation would think through what metadata columns and content types it needed at the very start of the implementation. The organisation would also put governance arrangements in place so that when teams need a content type for their document libraries they would be required to either choose an existing one or justify the need for setting up a new one.
  • SharePoint is asking content types to do several different things: to determine what metadata is captured about documents, to link retention rules to documents, and to specify a document template for documents. Sometimes there is a tension between these three roles. 
  • A content type only functions within the boundary of a site collection: if you have more than one site collection you will need to set up new content type for each site collection. (Read this post for an excellent description of the SharePoint site collection and the reasons why you may need more than one of them)

 

Alternative ways to apply retention rules within SharePoint 2007

Retention rules in SharePoint are called expiration policies. SharePoint can apply expiration policies to libraries: document libraries in team sites, or records libraries in the Records Centre.
You have two options. You can either:

  • Link your expiration policy to document libraries within team sites OR
  • Link your expiration policies to content types and route documents needed as records to the SharePoint records centre where they will inherit the retention rule of the content type

Linking expiration policies to document libraries

Linking your expiration policies to document libraries within SharePoint has the advantage that the policy is applied to documents in situ: you do not have to move them to a records centre.  However it does constrain teams to restrict document libraries to records sharing a common retention rule, as you can only apply one retention rule per document library.  Microsoft never intended document libraries to be used for storing records: they intended that documents needed as records would be sent to the records centre.  The expiration policy on the document library is merely to ensure that  documents not needed as records are disposed of promptly.  Document libraries do not protect documents against amendment or deletion.  

I have not yet heard of an organisation that has taken the route of attempting to store its records in team site document libraries and apply expiration (retention) policies to those document libraries.   The significant administrative overhead in implementing the records centre option may tempt some organisations to use this route.

Linking expiration policies to content types

Microsoft’s intended model for records management within SharePoint involves the use of content types to link retention rules to records and to route records to a records centre.  In the records centre documents are protected from unauthorised amendment and deletion and the retention rule is applied to them.  

The big disadvantage of applying expiration policies via content types is that SharePoint will only apply that expiration policy to those documents once they have been routed from documents libraries to the SharePoint records centre. This gives you the significant administrative overhead of ensuring that documents from that content type are moved to the right place in the records centre at the right time. 

 

Critical assessment of the SharePoint records centre

This is how Microsoft intended records management to work within SharePoint 2007 (out of the box, no customisation)

  • Expiration policies (retention rules) are defined
  • Content types are defined, and have expiration policies linked to them.
  • Document libraries are set up within team sites, with content types enabled
  • A records centre is set up within SharePoint
  • A records routing table is set up for each content type (this tells SharePoint where to locate documents of that content type when they are routed to the records centre)
  • Colleagues collaborate on documents within document libraries in team sites
  • When colleagues wish to save a document as a record they right click on it, and choose the option ‘send to records centre’.
  • When the option ‘send to records centre’ is chosen, SharePoint takes a copy of the document and sends the copy to the records centre. It routes the document to the records library within the records centre designated for that content type. In the records centre the document is protected from amendment or deletion for as long as the retention period specified for that content type

The fact that SharePoint forces us to move documents from the team site to the records centre in order to protect them and apply retention rules to them poses us the following major challenges:

  • How do we ensure that all or most documents needed as records are routed to the records centre?
  • How do we avoid losing all the context that the document had when it sat within the team site?
  • How do we ensure that the document is routed to the right place in the records centre?
  • What do we do when teams use methods other than documents to record their work?
  • How do we ensure that the records centre becomes a useful resource, that repays the resources expended in setting it up and maintaining it?

Ensuring all or most documents needed as records are routed to the records centre

Routing documents to the records centre depends upon colleagues right clicking on the document as it sits in their team site document library, and selecting the option ‘send to records centre’. The likelihood is that many colleagues will omit to take this step. Sending a copy of the document to the records centre adds no appreciable value to the team that created the document, who already have access to it within their team site.

DEFRA have tried an innovative approach of asking colleagues to indicate when they save a document whether a document is is of high, medium or low importance when they save it within a team site document library.   DEFRA have customised SharePoint to automatically route across to the record centre any document defined as high or medium importance that has not been changed for six months.

Avoiding loss of context when the document moves from the team site to the records centre

When a document is saved to a document library within a team site it possesses some context. That context is provided by the other activity that is taking place within the team site: the discussion forum, calendar, announcements, wikis, blogs, any other document libraries etc. It is also provided by the name of the team site itself, and by the navigation pathway to the team site. When a document is routed away from the team site to the records centre it loses most of that context.

Out of the box when a document is routed from a team site to the records centre it carries with it the metadata about the document itself, and the name of the document library that it sits in.    DEFRA customised the routing of documents to the records centre in order to ensure that a document additionally brings with it details of the navigation pathway down to the team site.

Ensuring that documents are routed to the right place within the records centre

Microsoft envisages that for every content type you create you have a designated records centre library ready to receive records of that content type. When a document is sent to the SharePoint records centre it is routed to whichever records centre library that content type has been linked to in the records routing table. One records centre library may be set up to receive records of more than one content type.

If a colleague sends a document to the records centre that is not allocated to a content type, or which is allocated to a content type that does not have a routing table set up for it, then the document is simply sent to an ‘unclassified’ library within the records centre.

Challenges arise because content types are best suited for document types (invoices, reports, contracts etc), whereas  the record of any particular piece of work (a project, a programme etc.) usually contains a variety of different document types.

Don Lueders has written this useful blog post describing how a client of his customised SharePoint so that they could set up record centre libraries for different types of work (rather than different types of document). The work around involves getting the records centre to determine which library to send the document to based on a metadata field within the content type rather than the content type itself: the metadata field in question can be set to capture the name of the piece of work.

Capturing records when teams use methods other than documents to record their work

Within a SharePoint team site, documents are just one of the means a team can use to record their work. They may alternatively choose to record an aspect of a piece of work with a wiki, a blog, a discussion forum. Information recorded within these outputs cannot be routed to a records centre. The records centre is only geared up to accept documents.

The most serious example of this relates to the ‘Document workspace’. The document workspace is a facility that allows a team to create a collaboration area around one particular document. This is useful where a document requires the input of several people, and it allows colleagues to have a discussion around that document within the workspace, as well as manage different versions of the document. SharePoint allows you to save the actual document produced in the document workspace to a document library and then route it to the records centre. SharePoint does not enable you to roll up the discussions that took place within the document workspace and send them to the records centre with the document.

Ensuring that the records centre becomes a useful resource, that repays the resources expended in setting it up and maintaining it

Records exist to satisfy the need of different stakeholders to understand a particular work. When we think back 20 years the hard copy file in a traditional filing system was trusted because it was the tool that the people who carried out the work used whilst they were working. In SharePoint the working tool is the team site, and my prediction is that most stakeholders in a piece of work are likely to get a better impression of that work by looking at the team site, than by looking at the selection of documents siphoned off from the team site to go to the records centre. I do not anticipate heavy usage of the records centre as a reference tool. Colleagues will have gained familiarity with team sites from working in their own team site, but will have little or no familiarity with the records centre. If the SharePoint records centre is not used as a reference tool, then it will not justify the considerable administrative effort necessary to sustain it.

 

Conclusions

It is important for the future of records management that we come up with an electronic records management model that enables us to capture records from a team without shackling them with a corporate file plan. I applaud Microsoft’s attempts to come up with such a model, but I don’t think that the SharePoint records centre is there yet. As it stands at the moment it is too fiddly, requires too much customisation and too much administrative effort to sustain, and yields too little information retrieval benefit.

It is hard enough governing team sites and document libraries within SharePoint. The records centre option means that we are additionally having to ensure that a records centre library is set up for each area of work, that the appropriate content type is set up and enabled for each document library in each team site, and that the routing table for each content type sends documents from the document library to the appropriate records centre library.

My prediction is that the majority of organisations that deploy SharePoint 2007 will not deploy the records centre. Those organisations that acknowledge a serious records management need will, if funds allow, use a third party system to apply classification and retention rules to records. Those organisations that do not do not acknowledge a serious records management need will concentrate their efforts on governing SharePoint to prevent SharePoint sprawl, and will live without the ability to apply retention rules (much as organisations have limped on with shared drives despite its lack of records management functionality).

Note:  most of the links from this post point to Don Lueders’ excellent SharePoint records management blog:  if you are intending to deploy the SharePoint records centre it is well worth reading through the archives of Don’s blog for explanations of how it works and tips on customisations.

7 thoughts on “Evaluation of records management in SharePoint 2007

  1. James,

    This is an excellent critique of Sharepoint 2007 from a records management perspective. The industry – represented by organisations such as RMS, SoA, etc – needs to lobby Microsoft so that they understand and react positively to our concerns.

    Sharepoint will not go away. On the contrary, it will become even more embedded into Office desktop applications. Microsoft have come on in leaps and bounds in terms of their understanding of records management principles and are much more open to having a dialogue with the industry, as evidenced for example by their records management discussion forum/blog. I believe our industry now needs to do more to formally engage with Microsoft so that our concerns are addressed.

    Regards,
    Eldin.

  2. James,

    I found your coments to be most enlightening. Anyone interested in implementing SharePoint – even with Records Centre, may want to consider RecordPoint – a module developed by Unique World (www.uniqueworld.com.au) that ‘fills in the gaps’ where SharePoint is less compliant with recordkeeping standards and guidelines.

    Regards

    Anthony

  3. Hi James,

    A good critique of MOSS 2007 and it bears out our own experiences. If Microsoft are serious about entering the RM market, they will have to address the limitations.

    Regards

    Steve

  4. Hi Eldin, hi Steve

    Thank you very much for your comments. I heard last year that representatives from various national archives around the world were planning to meet with Microsoft, but I haven’t heard whether the meeting happened or what came out of it.

    The problem for us as a profession is that if the weakness of the records management isn’t preventing them selling it, where is the incentive for them to improve it? Especially as the changes we would need woud require major rewrite of the software as opposed to minor tweaks. I think Microsoft will continue to leave it up to third party EDRM vendors to come up with products that plug into SharePoint and apply records management disciplines.

    I agree it was a positive step that Microsoft set up a blog in 2006 to explain the records management model of SharePoint 2007. It was a good blog, it attracted good comments (I know you left some for them Eldin) and Micorsoft did engage positively with those comments. But why did they close the blog down in March 2007, just four months after SharePoint 2007 launched? Surely if Microsoft were serious about records management within their product they would have kept it open to report on best practices as they emerged and answer questions about the product.

  5. James,

    Thanks for plugging my blog! My hit count has spiked through the roof over the last couple of days. It’s nice to know so many people have a real interest in SharePoint records management.

    This is a great post. Your readers will get a lot of useful information here. I’d like to add a couple of comments.

    You actually can assign Expiration Policies to Libraries outside of the Records Center. This is a nice feature because it allows you to apply disposition to your non-record material in collaborative spaces and ultimately apply retention to all of your SharePoint content, both inside and outside of the Records Center.

    Some of the concerns you mentioned, such as misclassifying records into the Records Center, are absolutely valid, but I don’t believe they are unique to SharePoint. These kinds of issues are common to all records management solutions and need to be resolved through policy, training and (ideally) process automation.

    Finally, I’m keeping my fingers crossed that a lot of your other concerns will be addressed in the next release of SharePoint, as I mentioned in my Office 14 Wishlist post. We’ll see…

    Cheers,
    Don

  6. Hi James – thanks for this. I think this is probably one of the clearest explanations of SP I have ever read. A very thoughful and thought provoking piece, partiularly as after rushing to implement SP we found ourselves in exactly the situation you describe.
    Do you have an explanation for the lack of case studies – is it because nobody has done it, or nobody is publicising what they have done?
    Regards
    Paul

  7. Hey, James.

    I haven’t dropped in for a while, but I wanted to let you know I have a new post up on my SharePoint Records Management blog that I thought your readers would be interested in.

    It’s an interview with Adam Harmetz. Adam is the Lead Program Manager for the SharePoint Document and Records Management engineering team at Microsoft. Adam talks about some of the new records management functionality in SharePoint 2010. Pretty interesting stuff. You can find the post here:

    http://sharepointrecordsmanagement.com/2009/12/16/sprm-speaks-with-adam-harmetz-of-microsoft/

    Hope everything is going well on your side of the pond!

    Don

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s