Is SharePoint a records management system? – podcast

Last Friday Brad Teed (CTO of GimmalSoft) and I discussed whether or not SharePoint could be regarded as a records management system. We recorded the discussion for the ECM Talk podcast series.

Click on the play button below to hear the podcast. If the play button is not showing in your browser (it needs Flash) then you can get the podcast from here or from i-Tunes (search on ‘ECM Talk’). The podcast is around 45 minutes minutes long.

Brad said that SharePoint 2010 could be regarded as a records management system with the caveat that it did not do things in the way that traditional records management systems did them.

I conceded that SharePoint 2010 had records management features (such as holding and applying retention rules, holding a hierarchical classification, locking documents down as records) but I did not think that these features were brought together in a coherent enough way to justify calling SharePoint a records management ‘system’.

SharePoint 2010 offered organisations two different approaches to records management – the in-place approach and the records centre approach. Brad and I described and critiqued these two different approaches . I said it was a choice between ‘a rock and a hard place’ because both approaches had serious drawbacks:

  • The in-place approach left records scattered around team sites under the control of local site owners without providing any reporting capability to give a records manager visibility over them all
  • The records centre approach had the advantage of bringing records together into one place that the records manager could control. However it brought with it the complexity of managing the routing rules necessary to get documents from SharePoint team sites to the record centre

Brad and I will be debating the issue of records management in SharePoint live at the SharePoint Symposium in Washington DC on 2 November 2011.

Musing Over MoReq2010- podcast series with Jon Garde

There are many differences between MoReq2010 and previous electronic records management certification regimes (including DoD in the US, TNA in the UK, the previous version of MoReq in Europe etc.)

MoReq2010 is different in:

  • its fundamental assumptions (it assumes that records are captured in many different systems within an organisation, rather than in one records system)
  • what it is fundamentally trying to do (specify the minimum set of things that any application needs to do to manage its records, rather than specify a general records system that does everything any organisation would want its record system to do)
  • the concepts it uses  (most notably the concept of an ‘aggregation’ replacing the concept of a ‘file’ )
  • the way it is structured (with the functionality grouped into ten services)
  • the fact that it will develop over time (with new modules being added to meet to meet specific needs)

I spoke to Jon Garde (the lead author of MoReq 2010) before his talk to the 2011 IRMS conference.  He said that it was impossible to do justice to all those changes within the confines of a 30 presentation speech.  I suggested a podcast series in which we could discuss in detail the different areas of MoReq2010, and the thinking behind them. We have called the podcast series MusingOverMoReq2010.

We have recorded three epsiodes so far – episode 1 discusses the general philosophy behind MoReq 2010. Episode 2 looks at the classification service, and the role of a classfication within a MoReq2010 compliant system.   Episode 3 looks at the disposal service and how retention rules are applied within a MoReq2010 compliant system

As well as discussing an aspect of MoReq2010, each episode also contains news of recent developements around MoReq2010. This is to allow for the fact that MoReq2010 will continue to develop as extension modules are written, and as the testing centres get set up and certification gets underway. The third section of the show is a ‘postbag’ section where we will discuss any questions you send us.

You can listen to it on the Musing Over MoReq2010 website.   You can subscribe to it from the i-tunes store (search on Musing over MoReq2010) or by getting your podcatcher to subscribe to this feed: http://musingovermoreq2010.com/MusingOverMoReq2010_feed.atom

10 questions on the current state of the ECM market

On Wednesday I recorded an ECM talk podcast in which I put  10 questions  about the current state of the enterprise content management market to Alan Pelz-Sharpe.

Click on the play button below to hear the podcast. If you don’t see the play button (it needs Flash) then you can get the podcast from here. The podcast is just over 50 minutes long.

The ten questions, and a flavour of some of Alan’s answers are given below – there is a lot more detail in the actual podcast itself

Why have HP bought Autonomy?

Alan said that most analysts were surprised at how much HP paid for Autonomy.  The best guess at what HP (a hardware company) wants to do with Autonomy (a software company) is that they may wish to create some kind of appliance which has Autonomy’s IDOL search engine already loaded onto it (a bit like the Google search appliance).  One thing that HP and Autonomy have in common is that they have both bought well-regarded electronic records management vendors (Tower and Meridio respectively), and done very little with them.

How hard have the ECM vendors been hit by the rise of SharePoint?

Alan said that the ECM vendors haven’t bit hit as hard as you might think. Their revenues are still rising, and most of them enjoy good relations with Microsoft.

How does EMC and Open Text compare with the bigger ECM vendors (Oracle and IBM)

Alan said that Oracle and IBM are so big because they do a huge variety of stuff as well as ECM.  But at the end of the day if you are buying FileNet from IBM you are dealing with the FileNet division, not the whole massive company. So for buyers of ECM systems company size doesn’t matter that much.  Open Text is the largest company that focuses exclusively on ECM.   EMC’s business is mainly about storage.  They bought Documentum, but Documentum is very different from the rest of the EMC group and there has not been many synergies.

What is happening in the CRM (Customer relationship management) arena and how does it relate to ECM?

Essentially ECM and CRM are seperate worlds without much overlap.  CRM is a vital tool for many organisations.  As yet there is not a great deal of tie-ins with ECM.  Oracle has both a CRM and an ECM suite, which work together reasonably well.  SAP signed a large deal with Open Text but there doesn’t seem to be a huge number of organisations using SAP together with Open Text products.  Many of the CRM tools will do a little bit of document management of customer related documents, but for the most part organisations will have CRMs that don’t talk to whatever ECM product(s) they have

The Europeans have just revised their electronic records management specification (MoReq2010).  When will the US records management standard DoD 5015 be revised (it was issued back in 2007)

Alan said he didn’t know of any plans to revise DoD 5015.  SharePoint drove a horse and cart through DoD 5015 because Microsoft made the decision to release a document management product that did not comply with it but had huge market success.  Alan said he didn’t see the point in revising it –  it was specifically tailored to US Government (DoD stands for Department of Defense) so some of the requirements are overkill for organisations in other sectors.

After the podcast it occured to me that there was no need for DoD 5015 to be revised.  MoReq2010 is the first of the electronic records management specifications to be extensible. Rather than revise DoD5015, if there were requirements specific to the US  (or to particular sectors in the US) that were not covered in the core requirements of MoReq2010 then a seperate module could be written to cover those requirements for vendors wishing to target their products to the US market.

What is happening in the intranet arena?

Alan said that nothing dramatic is happening in the intranet arena.  Some intranet makeover projects will have been hit by the economic downturn.  Alan can’t understand why some organisations want to use the same product to manage there external web-site and their intranet – to him they are fundamentally different things with different requirements.

Do you know any organisation that manages their e-mail well?

Alan said that of all the ECM implementations that he sees, the type that gives the quickest and most reliable return on investment is an e-mail archiving tool brought in to take stored e-mails off the mail servers.  I said I would like to see some of the e-mail archiving vendors apply for certification for their products under MoReq2010, so that buyers could be more confident of their ability to export e-mails out of their e-mail archive if they neeeded to.

What do you think of PAS 89?

PAS 89 will be a UK standard on enterprise content management, with a view to becoming an international standard.  Alan said PAS 89 was a good attempt to define the scope of enterprise content management, although it was hard to think of what an organisation would specifically use it for.

How does Alfresco compare with the proprietary ECM products 

Alan said that if we were talking about open source ECM products Nuxeo should be mentioned alongside Alfresco. Both of them are established, mainstream enterprise content management systems.  The main difference between them and the proprietary ECM products is the licensing model.

How does Google Apps compare with the established ECM products 

In terms of impact on the ECM market Alan is more interested in Box.Net than Google Apps.  Alan and James discussed the prospect of new start ups deciding not to set up shared drives and instead using services like Box.Net in the cloud to provide a relatively simple place for colleagues to store and share documents.

After the podcast it occured to me that for a good 15 years we have been wondering what would replace shared drives. Shared drives have survived so long because anything that could have replaced them for general document storage (EDRMS, SharePoint) has proved more complex than shared drives, and so shared drives retained their role as an uncomplicated, quick place to store documents.  From a users perspective something like Box.net is as simple to use as a shared drive, and has the advantage that folders and documents can be shared with people outside the team, and with people outside the organisation.   From an organisation’s information management point of view box.net is currently little better than a shared drive in terms of being able to apply retention rules and a records classification (though maybe if an ecosystem grows around Box.net someone could come up with a MoReq2010 compliant plug in for it- that would be interesting!)


The implications of MoReq2010 for records management practice

The DLM forum is having its triennial conference in Brussels this coming December.  I responded to the call for speakers with the following submission:

MoReq2010 can be seen as an attempt to ensure that the records management baby is not chucked out with the EDRMS bathwater.

The EDRM idea was solidly based in records management theory, but lost its market viability after 2008 thanks to the global economic downturn, the rise of SharePoint, and perceived problems with usability and user acceptability of EDRM systems and their attendant corporate fileplans.

SharePoint 2007 and 2010 both offer records management features, but neither offers a well-thought through records management model.  Most organisations with SharePoint implementations have not attempted to use records management features such as the SharePoint records centre.  Of those that are trying to use those features only a relatively small number will be able to impose sufficient governance to enable them to viably manage records in SharePoint.

For most organisations SharePoint will be a records management problem rather than a records management solution.  In a few years time more organisations will be saying ‘we need help managing the records that are scattered around our SharePoint implementation’ than will be saying ‘thanks to SharePoint content types we can now apply retention rules to records across our organisation’.

MoReq2010 doesn’t kill off the EDRM model (you can  use a MoReq2010 compliant system as an EDRM provided it complies with the plug-in modules for a user interface and a hierarchical classification), but it does not attempt to revive it either.

The fact that MoReq2010 is offering two alternative to EDRM, rather than just one, whilst continuing to support the EDRM model itself, indicates that the profession is not yet ready to commit its weight behind one single approach.  It also means that we are in a transition period, during which many records managers and consultants will be uncertain as to what approach to advise their organisations to take.

The two new approaches offered by MoReq2010 are ways of dealing with the ‘multiple repository problem’ – the fact that every organisation deploys numerous different applications to create, capture and store content and records.   EDRM systems rarely tackled that problem.  They typically relied on colleagues voluntarily declaring material into the EDRM as records, and there was rarely any incentive for colleagues to move documents out of a line of business application into an EDRM.

The back-end repository approach

The first of the two approaches is what I would call the back-end repository approach (I would like to call it repository-as-a-service but I fear you may mistake it for a cloud offering).  In this approach a MoReq2010 compliant system governs content captured in the multiple different applications of the organisation.  It governs either by taking that content out of those applications and storing it in the MoReq2010 compliant system, or by protecting and governing that content whilst it stays within those applications themselves.

This is an approach that vendors have been working on over the past five years – both EDRM/ECM vendors looking for ways to continue selling to customers who have chosen SharePoint instead of an EDRM, and e-mail archiving vendors looking to expand the scope of their archiving systems.  It is also compatible with service orientated architecture of IT departments, but no-one knows yet how it will play with moves to the cloud. MoReq2010 for the first time offers a certification regime for vendors taking this approach, giving the approach more gravitas and credibility, and offering buyers reassurance that their back end repository/archive will not in itself become a black hole from which it is hard to migrate records.

The back-end repository approach significantly changes the role of the records manager.  In EDRM implementations the records manager was interacting with users, training them, cajoling them, tackling change management challenges, and designing classifications that end-users would directly interact with   In the  back-end repository model the records manager has a different role – connecting legacy applications to the back end repository, and trying to ensure that no new application is deployed into the organisation unless it hooks into the back-end repository from day one.  The interaction with, and impact on, end-users will inevitably be reduced, but it is to be hoped it won’t be eliminated entirely.   It will still be important for end-users to be aware of whether or not a piece of content that they have contributed to a particular application has been captured by the back-end repository.

The in-application approach

The second of the two approaches is the addition of records management functionality to each  application deployed in the organisation so that these applications can manage their own records.

This is the approach that I sense the authors of MoReq2010 would like to see prevail in the world. They are well aware that every time a record moves from system to system it loses context, and that ideally records management metadata would be captured from the moment a record was first captured into an application.

This approach is beyond the capabilities of any single organisation – no organisation could customise all their applications for them to become MoReq2010 compliant.  It becomes viable only when the vendors of line of business systems, make their products MoReq2010 compliant – whether they be sector specific applications like social care systems for local authorities, or line of business applications like HR systems.  Its a battle worth taking on for the profession, and worth fighting, but success is likely to be patchy.  The hope is that a tipping point could be reached when everybody expected every application to be MoReq2010 compliant, and felt that something is wrong if it was not compliant.