Is there a sustainable and scaleable records management model in SharePoint 2010?

This afternoon I published the following article to the AIIM ERM community:

In June I saw Peter King, Office Server Group Manager of Microsoft UK, give a keynote speech to the AIIM roadshow in London. One of Peter’s slides contrasted functions that SharePoint 2010 could fulfill on its own, with functions that SharePoint needs the help of third party applications to fulfil. Records management was listed as being one of the things that SharePoint 2010 could do on its own.

However Microsoft say that they have no intention of submitting SharePoint 2010 for certification against one of the recognised standards for records management systems, for example DoD 5015.2 in the US or MoReq 2 in the European Union. They advise customers wanting that level of records management functionality to look to third party products.

SharePoint 2010 can do a form of records management, but it is records management as defined by Microsoft, not as defined by the international records management community. The fact that Microsoft are using their own definition of records management is not in itself a bad thing – we need innovation in this space, and vendors slavishly following worthy specifications drawn up by committee isn’t going to do anyone any favours. But it does mean that we need to closely scrutinise, and debate, the records management functionality contained in SharePoint 2010.

The lack of a clear records management model in SharePoint 2010

With SharePoint 2007 Microsoft had attempted to make two innovations in records management:

  • They wanted people to use ‘content types’, rather than folders as the primary means of organising and managing documents and other content
  • They wanted documents to be moved into a records centre to be treated as records, rather than treated as records inside the team sites that people worked in

In SharePoint 2007 there was a clear records management model – the only problem was it didn’t work. Microsoft had given all the power to the content type and none to the folder, but the content type on its own proved unequal and unsuited to the task of managing records.

In SharePoint 2010 Microsoft are hedging their bets:

  • Content types have been adapted to make them more suited to managing records (through the addition of a new kind of content type called a ‘document set’). But at the same time folders have been given more functionality
  • Record centres have been improved, but Microsoft are also providing an ‘in place’ records management facility to enable people to protect records from deletion or amendment without moving them to the records centre

In SharePoint 2010 there is no single records management model. Instead there are different choices and options. SharePoint 2010 shows some improvements in records management functionality when compared with SharePoint 2007. But the lack of a clear records management model in SharePoint 2010 means it will not be easy for implementers to decide upon and to sustain a coherent approach to records management.

For example lets imagine you intend to deploy a fileplan in SharePoint. I am defining a fileplan as a hierachical classification of the work an organisation carries out. There are three different ways you could deploy that fileplan. You could deploy it as:

  • the site collection/ site/ sub-site hierarchy
  • a hierarchical structure within a records centre (or spread across several records centres)
  • a taxonomy in the managed metadata service

Each of these options has its pros and cons.

Using the site collection/site/ sub site hierarchy to house your fileplan (‘In place’ records management)

Whereas in SharePoint 2007 you had to send a document to the records centre if you wanted it protected from unauthorised deletion or amendment, in SharePoint 2010 you can have it protected within the team site.

There is a ‘Declare Record’ button on the SharePoint document library ribbon that enables colleagues to save a document as a record. Once a document is declared as a record it is protected from amendment or deletion.

In theory this option means we can dispense with the records centre(s).

However having content protected as a record is only one aspect of how records managers want content needed as a record to be treated. We also want to to ensure that content is organised within a managed structure. A structure that persists, and continues to make sense, over time.

The problem with managing records ‘in-place’ is that it is very difficult to control the structure within which content resides.

Take a typical piece of content in SharePoint – a document, sitting in a folder within a document library. The library sits within a sub-site, which sits within a site, which sits within a site collection.

The structure that a document resides in is the breadcrumb trail encapsulated in the URL of the site, the library and the document itself . The breadcrumb trail starts with the name of the site collection and carries on with the name of the site, sub-site, document library and folder.

In theory you could impose the rigidity of a hierarchical records fileplan onto this structure. It is possible to:

  • set up a SharePoint site collection for each of the top level headings in your fileplan
  • set up a site within those site collections to represent the 2nd level of your fileplan, and sub-sites beneath that for the third level of the fileplan. If you needed another level of hierarchy you would add another level of sub-sites
  • ask each team to create a new document library for each new piece of work that they start, within a lowest level sub-site

However in practice such rigidity may become too much of straitjacket for a SharePoint implementation. If SharePoint becomes popular in your organisation then sub-sites and document libraries will be created for all sorts of reasons, some with the intention of holding records, but others for ephemeral collaboration. It isn’t necessarily going to be meaningful to shoe-horn them all underneath a fileplan hierarchy.

On the other hand if you do not think through the hierarchy of site collections/sites/sub-sites, and allow a free for all in site and sub-site creation, then you end up with a haphazardly evolving structure, a sprawl. When that happens the in-place records management feature is of little use, because your declared records are randomly dotted around the massive SharePoint sprawl like tiny islands in the Pacific Ocean.

Housing your fileplan in a SharePoint records centre

The way the records centre works in SharePoint 2010 is broadly similar to the way that it worked in SharePoint 2007 (although to say that the records centre worked at all in SharePoint 2007 is in my view something of an exaggeration).

  • In SharePoint 2007 the idea was that knowledge workers would click on a document in a SharePoint document library, and would select the option ‘send to the records centre’. In the records centre the ‘records routing table’ would route the document to the appropriate records library, providing the document belonged to a SharePoint content type, and provided that a records routing rule had been set for that content type.
  • In SharePoint 2010 the records routing table is replaced by the content organiser and the drop off library. When someone selects the option ‘send to records centre’ documents are received by the drop off library, and then processed by the content organiser. The content organiser sends the document to the appropriate records centre library, or the appropriate folder within a records centre library, providing that the document belongs to a content type or a document set, and providing that a routing rule has been set for the content type or document set that the document belongs to

In terms of configuration in SharePoint 2010 you can have a drop off library and content organiser within the records centre itself (like the records routing table in SharePoint 2007). You can also have content organisers and drop off libraries inside each collaborative site, that route documents to the records centre or to particular places within the records centre.

The improvements made to the records centre between SharePoint 2007 and SharePoint 2010 are incremental rather than revolutionary, but they are worth listing:

  • Routing to a specific folder – In SharePoint 2007 the fact that you could have a folder structure within a records centre library was rendered useless by the fact that the records routing table could only route a document to a records library, it could not route a document to a specific folder within a records library. In SharePoint 2010 the content organiser can route a document to a specific folder within a records library in the records centre
  • Inheritance from folders– Folders have more functionality in SharePoint 2010 (they had none in 2007). You can set retention rules on high level folders, and you can allow all the sub-folders and documents saved within it to inherit the retention rule. This means you can get a folder structure in a records centre library to function like a fileplan
  • Routing on the basis of metadata properties In SharePoint 2007 the routing table could only apply a rule to a content type. In 2010 rules can be set rules at a more granular level. You can base a rule on a value in a metadata column field of a content type. For example you could define a column entitled ‘project’ for a content type/document set. You could set up a controlled vocabulary for the ‘project’ metadata column. You could get the content organiser to look at the value in that column for any incoming document, and route it to the folder in the records centre for that particular project.
  • Multiple records centres – In SharePoint 2007 organisations were only able to have one records centre. In SharePoint 2010 you can have as many records centres as you wish. Rather than having one records centre with one huge fileplan, an organisation could choose to spread its fileplan over several records centres in SharePoint 2010

The whole concept of the record centre in SharePoint brings with it three big disadvantages

  • It requires you to use content types – If you are using the records centre, you are going to have to use content types or document sets. This is because the content organiser works by applying rules to content types/document sets, not to folders. Content types and document sets are more powerful than folders, but they are also more complex for you as an administrator to set up and maintain, and for your colleagues to understand and use.
  • You have the administrative overhead of maintaining all the routing rules – Most organizations are used to having some rules to automatically route some documents (particularly where they have workflows set up to automate some important business processes, or within some line of business systems). However I don’t know any organisation that is used to setting rules for the routing of documents across all its business processes. This reliance on pre-defined routing rules for the basic records management process is new to SharePoint – none of the established electronic records management system worked that way. I am yet to be convinced that it is feasible to set up and maintain routing rules covering a large organisation’s entire range of activities
  • there is no guarantee that the records in the records centre will be useful– Microsoft envisages collaboration sites as being the sphere of the knowledge worker, and records centre sites as being the sphere of the records manager. But if knowledge workers have little or no interaction with the records centre, then in what sense can we regard the accumulation of documents that build up there as an authentic record of the work that those knowledge workers carry out?

Installing your fileplan as a taxonomy in the Managed Metadata service

The third option for your fileplan in SharePoint 2010 is to have it as a controlled vocabulary, a three level taxonomy that sits in the Managed Metadata Service, and that can be used to:

  • populate metadata columns in document libraries
  • and/or

  • populate columns of metadata for content types and/ document sets

Support for hierarchical taxonomies is a new feature in SharePoint 2010. SharePoint 2007 only supported flat controlled vocabularies (akin to a drop-down list of headings).

A taxonomy in SharePoint’s managed metadata service can only go three levels deep (the three levels are called: groups, term stores and terms). This is a significant constraint – in my experience those organisations of around 800 people that have corporate fileplans tend to have a four level classification (which they use to organise folders that are created when a piece of work starts).

The weakness of having your filepelan as a taxonomy in the managed metatdata service is that you cannot link retention rules to SharePoint taxonomy terms. You can only link retention rules to content types, document sets, libraries and folders. You would still need to decide with this option whether to move records to the records centre, or whether to manage records in-place in the collaborative team sites.

If you are using the in-place records management option, then the advantage of having the fileplan as a taxonomy is that it removes the need to warp the site collection/site/sub-site structure into the form of a fileplan.

It is not easy to enforce a fileplan set up as a SharePoint taxonomy, particularly with the in-place records management option. This is because SharePoint gives teams a lot of options on what type of object to use when collaborating on work. It is hard to predict whether any particular team will want a sub-site, a document library, or a folder for their piece of work. This makes it harder to issue a blanket policy rule such as ‘every document library must be classified against the fileplan in the managed metadata service’

One option might be to use content types, and specifically to use the SharePoint document set content type. In theory it is possible for an organisation to use ‘document set content types’ as a fourth level fileplan classification, with each ‘document set content type’ linked to a specific pathway in the three level fileplan in the managed metadata service. The downside of this approach is that it would force people to use document sets to manage documents of their work. Document sets are a new feature in SharePoint 2010, and at the time of writing I have not seen any case studies from organizations using them in practice.

Conclusion

SharePoint is now a mature collaboration system, having gone through several iterations of the product. But the records management in SharePoint is not yet mature. The records centre in SharePoint 2007 could not be made to work to any scale without extensive custom coding. Many of the concepts on which records management is based in SharePoint (content types, document sets, routing rules, and the records centre) are unique to SharePoint. Records management in SharePoint is still unproven, and will remain unproven until we know of a reasonably sized organisation who have successfully used it for records management across their full range of activities without extensive customisation.

For these reasons I think it is important that we as a records management community look on SharePoint 2010 with a sceptical, questioning eye.

7 thoughts on “Is there a sustainable and scaleable records management model in SharePoint 2010?

  1. A good piece, however I think it misses a key point which is that traditional EDRMS cannot be AFFORDED by most organisations. And rarely enable a good business case to be developed as they do not produce tangible, cashable, business benefits.

    So to me many businesses don’t have a choice, as SharePoint does provide lots of benefits at relatively low cost compared to old style EDRMS. Record Management is a luxury many cannot afford. IMHO!

    1. Martyn has made a very valid point IMHO. I currently work with an EDRMS solution which works extremely well for EDM and adequately for ERM. But the market-leader software that drives it has proved expensive. As Martyn says, many organisations simply cannot afford such solutions. It is inevitable therefore, that many will opt for a cheaper alternative – Sharepoint – and will compromise on the traditional RM catchism. The RM community needs to adjust its aspirations from a ‘one-size-fits-all’ model to ‘what’s appropriate and affordable for us’.

      We all love to criticise M$ for its blunderbuss approach, which is justified in many cases. But at least, it makes us re-think our assumptions about how things should be in RM.

      All things evolve, even RM.

      1. Is it really cheaper to implement SharePoint though? Off the shelf cheaper perhaps, but anecdotally I’ve heard that it costs considerable dollars to implement and maintain. Would love to hear comments from people who have implemented it regarding this.

        Cheers

  2. James thanks for the interesting post. I hear your concerns regarding the approach to implementing a records management solution/program within SharePoint 2010. However as we are part of a records management community that is no longer “stuck” in the archives room in a basement somewhere (we are now part of the everyday life of a user) we need to look at this through different lenses…

    We are approaching today’s content creation and management challenges with a traditional records management ego, and this is causing a lot of tension, we are seen as being reluctant to adopt and adapt new ways of working and we are getting nowhere. I work with so many organisations that still in the year 2010 have a mixture of file system document storage, portal or intranet, records management system and collaboration or content management system. I feel that the records management profession has a tremendous amount of value based on our learning’s that we can bring to solving todays Information challenges and our success is related to blending our approaches with today’s users and technologies.

    Your blog describes many of the new features in SharePoint 2010 and I would like to add some additional comments around these topics:

    1)The lack of a clear records management model in SharePoint 2010

    Yes there are various models one can adopt; however you do not need to only select one. We finally have a solution that gives us choices. Remember the typical eDRMS project, one choice and one choice only, create records in Office , store in a eDRMS and File Plan etc…, resulting in poor user adoption, and the only thing compliant about that was the technology.

    The options within SharePoint 2010 means that we can implement a solution that addresses more than just one way of working, we have users with varied requirements (largely they have no compelling reason to create records) and maturity and need to be flexible in how we deliver an outcome for them. The important thing with SharePoint and 2010 is to have a PLAN for how RM will be implemented and managed, as people adopt SharePoint just as they plan for migrating content in they need to plan how this will be managed as records, where it will be managed and who will manage it.

    2.Using the site collection/site/ sub site hierarchy to house your fileplan (‘In place’ records management)

    Adopting a Site Hierarchy approach for housing the File Plan is something I would stay right away from. This will force traditional approaches for records management but destroy the collaborative nature of SharePoint.

    In 2010 the metadata navigation and filtering can provide records managers with the context, its centralised, has management and is navigable. So in place records can be used with this functionality to give the record manager the user experience that they are familiar with. In place records in my opinion should not be used to manage all records, but records relating to temporary records, not VITAL or PERMANENT records. Do not clutter the collaboration structures with content that will be there for long periods, but DO MANAGE it.

    3)Housing your fileplan in a SharePoint records centre

    I am going to provide some greater insight into the disadvantages you identified:
    •We don’t (only) have to use multiple CT types in conjunction with the records centre. I can classify records in the File Plan using the standard “Document” content type with a simple content organiser rule
    o Of content type “document” add a simple asterisk * in the rule to allow all other content types to be evaluated and filed in the correct File Plan hierarchy
    o Create a new folder for each new term that the content organiser evaluates
    o Each top level folder can have a retention period assigned through the “compliance details” and inherited by all sub folders and records added to the File plan
    o Each new record within the file plan automatically inherits the retention period
    •Doc Sets a hugely powerful and great for managing Case Type records e.g. Projects, Recruitment, Auditing etc….., create a syndicated document set content type that is shared through the enterprise content hub and easily available to other site collection , yet centrally maintained.

    I won’t add more configuration options, if people want to know more please drop me an email and I will answer any specific questions……

    4) Installing your fileplan as a taxonomy in the Managed Metadata service
    The implementation of the File plan as a managed metadata service is to aid classification at the point of capture and where possible I recommend the use of auto classification via folder, library or content type metadata. There are various techniques we can use in SharePoint to ensure classification occurs at the point of creation , the good thing about this is we don’t need to get a user involved….and we have the tools to provide a lot of automation in this regard.
    When documents are classified at creation, and turned into a record then this classification metadata assists in ensuring the record is classified correctly in the File Plan. A reduced burned on users who do not have to add classification metadata and get it wrong.

    Conclusion
    I feel that looking at anything with a skeptical eye, only clouds one’s ability to see beyond what we know and are comfortable of whilst missing the opportunity to open ourselves up to adopting and adapting to change. As records management professionals we need to constantly be looking for ways in which we can get greater buy in to what we are trying to achieve and we have the knowledge to apply this to the SharePoint environment. We just have to be willing to accept that as the users and organisation need to change so do we.

  3. Hi James,

    Thanks for this really useful overview of what Sharepoint 2010 can/can’t do. I have a few thoughts based on it:

    1)If we use In Place RM rather than records centres, is there any way of configuring SP so that once the ‘Declare as Record’ button is actioned, it triggers the addition of metadata (via the Managed Metadata function) to indicate that it is now an official record? I am wondering if this would be a way around the limitations of not having the records captured in an organised structure. Presumably we then have the capability to run a search for all declared records for (e.g.) HR functions. This would also assist in putting together a digital preservation plan to ensure that all records held long term or permanently are properly managed to ensure ongoing accessibility.
    2)Managed metadata and keywords sounds like we are moving towards tagging and folksonomies which I don’t think is a bad thing. I like the user having the option to tag docs which would hopefully promote a greater sense of ownership of the document. If it can be linked into the folder structure it should also aid search and retrieval capability.
    3)Content type sounds rather fiddly to configure, but can essentially be used to create templates for key documents to ensure metadata is captured at the point of creation of the document. This is potentially of great value to my company as it would not require the user to manually fill out a metadata table each time they create a new doc (although they can tag it with keywords if they require). This will also tie in with our Protective Marking policy by making it easier for the user to classify and identify confidential docs.
    4)If we use content types, we can then use the Records Centre (rather than in place RM) but also with the support of Managed Metadata as a search aid.
    5)You say that there is “no guarantee that the records in the record centre will be useful…” – this is a good point but can be tackled through training, governance and ongoing support. Given the quantities of information that my company produces, it is inevitable that the user will have to take responsibility for declaring what is or isn’t a record
    6)Summing up, it looks like SP2010 offers an opportunity to redefine records management to the more fluid way in which records and information are created and consumed these days, and whilst it does throw up challenges in terms of configuration and ongoing administration, we need to be more flexible to keep up

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s