The noughties was a decade of two halves for electronic records management systems in the UK

The first half of the decade: 2000 to 2005

The first half of the decade was a boom period. The UK National Archives (then called the Public Record Office) issued an influential statement of requrements (TNA 2002) for electronic records management systems, with an accompanying evaluation programme. Similar evaluation programmes were set up in other countries. The requirements included the ability to:

• hold a hierarchical corporate records classification (fileplan)
• link retention rules and access rules to the fileplan
• allow folders and documents to inherit retention and access rules from the heading of the fileplan that they are saved under

The acronym EDRMS (electronic document and records management systems) was applied to systems that complied with statements of requirements such as TNA 2002. In the UK large organisations in the public sector and in regulated sectors of the economy would routinely specify TNA 2002 compliance when issuing a tender for a corporate scale document management system.

Like most booms it was overdone. Some UK central government departments rushed into implemementations, spurred on by a Modernising Government target (all UK government departments must create new records electronically by 2004), and by a desire to be ready for the coming into force in January 2005 of the Freedom of Information regimes for England and Wales, and for Scotland. Corporate Fileplans proved difficult for organisations to construct and unpopular for users to use.

Nevertheless EDRMS had established itself as an integral part of the enterprise content management (ECM) landscape. All the big ECM suite vendors (IBM, Open Text, ECM, Oracle etc.) now provide an EDRMS offering within their suites, alongside their web content management, portal and workflow offerings.

The second half of the decade – 2005 to 2009

In the second half of the decade EDRMS has struggled. Some of the factors behind the decline are specific to the UK, but the more important ones are global. The Modernising Government target deadline passed unnoticed, and the fear of Freedom of Information subsided. The National Archives withdrew their testing regime at the end of 2005 to make way for a European Union sponsored regime (MoReq2). It would be three years before the MoReq 2 specification would be published, and by that time EDRMS was facing a new threat in the shape of SharePoint 2007.

SharePoint 2007 came out towards the end of 2006. It had no facility to apply a corporate fileplan, and an obscure and unworkable method for applying retention rules (in the shape of the SharePoint Records Centre). SharePoint 2007 devolves power down to the team level, and hence did not hit the same user adoption problems that EDRMS implementations often ran up against. Microsoft cleverly/generously included client licences for SharePoint in the Enterprise Agreements that most organisations that buy Windows, Exchange and Office already have with them. This meant that many organisations wanting corporate wide document management systems did not issue an invitation to tender (ITT): instead they simply started using the SharePoint licences that they already had. This has had a ruinous effect upon electronic records management system evaluation programmes, such as MoReq2, whose influence is predicated upon buyers specifying compliance when they issue an ITT.

So here we are at the start of 2010. The acronym EDRMS has almost disappeared from our language. What are the factors that will shape the way that the electronic records management system market develops over the coming years?

Make or break for MoReq2

MoReq 2 came out at a bad time for records management standards and evaluation programmes. The double whammy of the global downturn and the rise in SharePoint meant that there was less money to be made out of electronic records management systems. In order to comply with MoReq2 all vendors, even those meeting existing standards such as TNA 2002, would need to make investments in their products to make their necessary changes. Most of the vendors decided not to make those investments and have not yet submitted their products for testing. The result is that at the time of writing only one vendor (Fabasoft) has acheived compliance with the standard. In November 2009 I heard Doug Miles of AIIM tell an AIIM trade members meeting that so long as there was only vendor on the list then buyers would be unlikely to specify MoReq 2 compliance on their Invitations to Tender, because if they did so they would end up with a shortlist of one supplier.

As a result of this situation MoReq2 will be overhauled during the early part of 2010, with a view to making it less onerous for vendors to adapt their products. Some functionality that is currently labelled as mandatory will be downgraded to optional.

Rory Staunton sits on the MoReq2 governance committee, and he has written an interesting article in the January 2010 Records Management Bulletin. He does not pull any punches in the article.
He says:

The market reality is that records management software as a component market has almost collapsed. The sales of records management software licences from vendors have been falling before and during this recession, and most vendors are chasing a non-existent goal they call eDiscovery. US style eDiscovery is almost non existent in Europe, and MoReq2 in its original version was a basket case

Electronic records management has a tough battle ahead of it. Staunton believes that the best chance of MoReq 2 having an influence is to pursuade regulators to adopt MoReq 2

The way to win a game, of course, is to write the rules, so clever national archives, vendorrs and large users should target regulators in all sectors and propose that they adopt and extend MoReq2 based approaches…..The real success of MoReq2 will be judged in five years time, depending on the number of regulators that embracee and extend it.’

Staunton is thinking or regulators in fields ‘from envirnoment to nuclear, healthcare to telcos, retail to finance’

Staunton went on to say:

‘there is not time to waste if the records management community are to ride the wave of impending business regulations that are sure to emerge from the current and widespread breakdown in financial sector regulation. In market terms, Europe needs some basic approach to securing business information. Europe needs a more productive approach than the knee-jerk, lawyer-led, e-mail centric eDiscovery systems that the North Americans are adopting, which are a retrospective friction on business, and do not enhance compliance as they cure, rather than prevent, breaches of process’

The coming of SharePoint 2010

John Newton (chairman and CTO of Alfresco, a competitor to SharePoint) paints a bleak picture of the effect of SharePoint on ECM vendors, in a blogpost outlining his predictions for 2010:

You have to hand it to Microsoft; they seem to have scared the bejesus out of the traditional ECM vendors. They have all seemed to rolled over and played dead in the wake of SharePoint 2007

And it could get worse before it gets better. SharePoint 2010 is coming along with ‘ ever tighter integration with office, better web support, new records management, and claims to better scalability and administration’

Nevertheless Newton believes that SharePoint still has vulnerabilities, arising from the fact that organisations adopting it are tied into the whole Microsoft stack :

”Microsoft has chosen not to address its fundamental architectural flaw of storing content in the database and it is still an exclusively Microsoft-centric platform. Forget whatever database, operating system, language, browser you have – you better get used to SQL-Server, .NET, Windows, IE and don’t forget Silverlight. If the traditional vendors can’t battle the crap out of that, then they deserve to lose. The next 12 months will be critical during the transition to SharePoint 2010”

ECM and SharePoint: competition or partnership?

The reaction of ECM vendors to SharePoint 2007 has gone through 3 phases:

• At first ECM vendors believed that the difficulties of governing information in SharePoint 2007 would mean that SharePoint would not greatly threaten their market share.
• When it became clear that SharePoint 2007 was shipping at a rapid rate the ECM vendors started developing ‘web parts’ that could be deployed within the SharePoint environment. The web parts enabled people to save docutments to the ECM, edit them and search for them, without leaving their SharePoint environment.
• More recently vendors have realised that a much deeper integration with SharePoint is required. The problem with using web parts for integration with an ECM was that SharePoint knew nothing of the documents. Clients wanted the content to be available to SharePoint so that they had the option of using SharePoint functionality (such as workflows) with the documents and other content that they created.

Most of the ECM vendors that I have spoken to recently are adopting an approach to integration with SharePoint, 2007 which sees content stored in the ECM system, but made available to SharePoint so that to all intents and purposes SharePoint views it as SharePoint content

The ECM vendors have a reasonable tale to tell. Their selling proposition is:

• that you can use the ECM to house your fileplan, and to apply it to any system that you connect it to (not just SharePoint)
• you can apply the fileplan to any type of content (not just documents, but also content such as discussion board posts, blog posts etc.), and any level of granularity (not just documents, but folders of documents, document libraries or entire team sites)
• that they provides a more efficient model for storing content. SharePoint stores both the document/content itself and the metadata about it, in a database (SQL Server). ECM systems store the document/content in a repository, with only the metadata stored in a database
• that although the content is stored outside of SharePoint/SQL server, the content is made available to SharePoint so whatever you wanted to do in SharePoint with that content you would still be able to do.

However the vendors are facing an uphill battle. The problems for them are that:

• Most of the ECM vendors seem to have settled on the same model. They are fighting for a relatively small market, and customers may find it hard to differentiate them from each other.
• they are reacting to SharePoint, rather than causing problems for Microsoft. It is one way traffic. There is nothing that any ECM vendor has done over the past few years that Microsoft has felt compelled to react to. Microsoft has nothing to fear from the ECM vendors in the collaboration space. (Note that things are slightly different in the web content management and portal spaces where SharePoint has had less impact on the market share of traditional ECM vendors).
• As Microsoft gradually improve information governance within SharePoint they will leave less and less space for ECM systems in the collaboration space.
• ECM vendors will be reluctant to rock the boat with Microsoft. Several ECM vendors have told me that Microsoft has been very helpful to them, sharing their roadmap and plans for 2010 well in advance so that the ECM vendors can plan how their products will integrate with it. But it means that in the collaboration space ECM vendors are becoming partners rather than competitors with Microsoft.

It is interesting to note that the only two vendors that are trying to take Microsoft on in the collaboration space (Alfresco and Google) are not traditional ECM vendors. Alfresco provides open source collaborative and document management software. Google provides Google Apps from the cloud, as a subscription service.