Thinking Records

This afternoon I published the following article to the AIIM ERM community:

In June I saw Peter King, Office Server Group Manager of Microsoft UK, give a keynote speech to the AIIM roadshow in London. One of Peter’s slides contrasted functions that SharePoint 2010 could fulfill on its own, with functions that SharePoint needs the help of third party applications to fulfil. Records management was listed as being one of the things that SharePoint 2010 could do on its own.

However Microsoft say that they have no intention of submitting SharePoint 2010 for certification against one of the recognised standards for records management systems, for example DoD 5015.2 in the US or MoReq 2 in the European Union. They advise customers wanting that level of records management functionality to look to third party products.

SharePoint 2010 can do a form of records management, but it is records management as defined by Microsoft, not as defined by the international records management community. The fact that Microsoft are using their own definition of records management is not in itself a bad thing – we need innovation in this space, and vendors slavishly following worthy specifications drawn up by committee isn’t going to do anyone any favours. But it does mean that we need to closely scrutinise, and debate, the records management functionality contained in SharePoint 2010.

The lack of a clear records management model in SharePoint 2010

With SharePoint 2007 Microsoft had attempted to make two innovations in records management:

• They wanted people to use ‘content types’, rather than folders as the primary means of organising and managing documents and other content
• They wanted documents to be moved into a records centre to be treated as records, rather than treated as records inside the team sites that people worked in

In SharePoint 2007 there was a clear records management model – the only problem was it didn’t work. Microsoft had given all the power to the content type and none to the folder, but the content type on its own proved unequal and unsuited to the task of managing records.

In SharePoint 2010 Microsoft are hedging their bets:

• Content types have been adapted to make them more suited to managing records (through the addition of a new kind of content type called a ‘document set’). But at the same time folders have been given more functionality
• Record centres have been improved, but Microsoft are also providing an ‘in place’ records management facility to enable people to protect records from deletion or amendment without moving them to the records centre

In SharePoint 2010 there is no single records management model. Instead there are different choices and options. SharePoint 2010 shows some improvements in records management functionality when compared with SharePoint 2007. But the lack of a clear records management model in SharePoint 2010 means it will not be easy for implementers to decide upon and to sustain a coherent approach to records management.

For example lets imagine you intend to deploy a fileplan in SharePoint. I am defining a fileplan as a hierachical classification of the work an organisation carries out. There are three different ways you could deploy that fileplan. You could deploy it as:

• the site collection/ site/ sub-site hierarchy
• a hierarchical structure within a records centre (or spread across several records centres)
• a taxonomy in the managed metadata service

Each of these options has its pros and cons.

Using the site collection/site/ sub site hierarchy to house your fileplan (‘In place’ records management)

Whereas in SharePoint 2007 you had to send a document to the records centre if you wanted it protected from unauthorised deletion or amendment, in SharePoint 2010 you can have it protected within the team site.

There is a ‘Declare Record’ button on the SharePoint document library ribbon that enables colleagues to save a document as a record. Once a document is declared as a record it is protected from amendment or deletion.

In theory this option means we can dispense with the records centre(s).

However having content protected as a record is only one aspect of how records managers want content needed as a record to be treated. We also want to to ensure that content is organised within a managed structure. A structure that persists, and continues to make sense, over time.

The problem with managing records ‘in-place’ is that it is very difficult to control the structure within which content resides.

Take a typical piece of content in SharePoint – a document, sitting in a folder within a document library. The library sits within a sub-site, which sits within a site, which sits within a site collection.

The structure that a document resides in is the breadcrumb trail encapsulated in the URL of the site, the library and the document itself . The breadcrumb trail starts with the name of the site collection and carries on with the name of the site, sub-site, document library and folder.

In theory you could impose the rigidity of a hierarchical records fileplan onto this structure. It is possible to:

• set up a SharePoint site collection for each of the top level headings in your fileplan
• set up a site within those site collections to represent the 2nd level of your fileplan, and sub-sites beneath that for the third level of the fileplan. If you needed another level of hierarchy you would add another level of sub-sites
• ask each team to create a new document library for each new piece of work that they start, within a lowest level sub-site

However in practice such rigidity may become too much of straitjacket for a SharePoint implementation. If SharePoint becomes popular in your organisation then sub-sites and document libraries will be created for all sorts of reasons, some with the intention of holding records, but others for ephemeral collaboration. It isn’t necessarily going to be meaningful to shoe-horn them all underneath a fileplan hierarchy.

On the other hand if you do not think through the hierarchy of site collections/sites/sub-sites, and allow a free for all in site and sub-site creation, then you end up with a haphazardly evolving structure, a sprawl. When that happens the in-place records management feature is of little use, because your declared records are randomly dotted around the massive SharePoint sprawl like tiny islands in the Pacific Ocean.

Housing your fileplan in a SharePoint records centre

The way the records centre works in SharePoint 2010 is broadly similar to the way that it worked in SharePoint 2007 (although to say that the records centre worked at all in SharePoint 2007 is in my view something of an exaggeration).

• In SharePoint 2007 the idea was that knowledge workers would click on a document in a SharePoint document library, and would select the option ‘send to the records centre’. In the records centre the ‘records routing table’ would route the document to the appropriate records library, providing the document belonged to a SharePoint content type, and provided that a records routing rule had been set for that content type.
• In SharePoint 2010 the records routing table is replaced by the content organiser and the drop off library. When someone selects the option ‘send to records centre’ documents are received by the drop off library, and then processed by the content organiser. The content organiser sends the document to the appropriate records centre library, or the appropriate folder within a records centre library, providing that the document belongs to a content type or a document set, and providing that a routing rule has been set for the content type or document set that the document belongs to

In terms of configuration in SharePoint 2010 you can have a drop off library and content organiser within the records centre itself (like the records routing table in SharePoint 2007). You can also have content organisers and drop off libraries inside each collaborative site, that route documents to the records centre or to particular places within the records centre.

The improvements made to the records centre between SharePoint 2007 and SharePoint 2010 are incremental rather than revolutionary, but they are worth listing:

• Routing to a specific folder – In SharePoint 2007 the fact that you could have a folder structure within a records centre library was rendered useless by the fact that the records routing table could only route a document to a records library, it could not route a document to a specific folder within a records library. In SharePoint 2010 the content organiser can route a document to a specific folder within a records library in the records centre
• Inheritance from folders- Folders have more functionality in SharePoint 2010 (they had none in 2007). You can set retention rules on high level folders, and you can allow all the sub-folders and documents saved within it to inherit the retention rule. This means you can get a folder structure in a records centre library to function like a fileplan
• Routing on the basis of metadata properties In SharePoint 2007 the routing table could only apply a rule to a content type. In 2010 rules can be set rules at a more granular level. You can base a rule on a value in a metadata column field of a content type. For example you could define a column entitled ‘project’ for a content type/document set. You could set up a controlled vocabulary for the ‘project’ metadata column. You could get the content organiser to look at the value in that column for any incoming document, and route it to the folder in the records centre for that particular project.
• Multiple records centres – In SharePoint 2007 organisations were only able to have one records centre. In SharePoint 2010 you can have as many records centres as you wish. Rather than having one records centre with one huge fileplan, an organisation could choose to spread its fileplan over several records centres in SharePoint 2010

The whole concept of the record centre in SharePoint brings with it three big disadvantages

• It requires you to use content types – If you are using the records centre, you are going to have to use content types or document sets. This is because the content organiser works by applying rules to content types/document sets, not to folders. Content types and document sets are more powerful than folders, but they are also more complex for you as an administrator to set up and maintain, and for your colleagues to understand and use.
• You have the administrative overhead of maintaining all the routing rules – Most organizations are used to having some rules to automatically route some documents (particularly where they have workflows set up to automate some important business processes, or within some line of business systems). However I don’t know any organisation that is used to setting rules for the routing of documents across all its business processes. This reliance on pre-defined routing rules for the basic records management process is new to SharePoint – none of the established electronic records management system worked that way. I am yet to be convinced that it is feasible to set up and maintain routing rules covering a large organisation’s entire range of activities
• there is no guarantee that the records in the records centre will be useful- Microsoft envisages collaboration sites as being the sphere of the knowledge worker, and records centre sites as being the sphere of the records manager. But if knowledge workers have little or no interaction with the records centre, then in what sense can we regard the accumulation of documents that build up there as an authentic record of the work that those knowledge workers carry out?

Installing your fileplan as a taxonomy in the Managed Metadata service

The third option for your fileplan in SharePoint 2010 is to have it as a controlled vocabulary, a three level taxonomy that sits in the Managed Metadata Service, and that can be used to:

• populate metadata columns in document libraries

and/or

• populate columns of metadata for content types and/ document sets

Support for hierarchical taxonomies is a new feature in SharePoint 2010. SharePoint 2007 only supported flat controlled vocabularies (akin to a drop-down list of headings).

A taxonomy in SharePoint’s managed metadata service can only go three levels deep (the three levels are called: groups, term stores and terms). This is a significant constraint – in my experience those organisations of around 800 people that have corporate fileplans tend to have a four level classification (which they use to organise folders that are created when a piece of work starts).

The weakness of having your filepelan as a taxonomy in the managed metatdata service is that you cannot link retention rules to SharePoint taxonomy terms. You can only link retention rules to content types, document sets, libraries and folders. You would still need to decide with this option whether to move records to the records centre, or whether to manage records in-place in the collaborative team sites.

If you are using the in-place records management option, then the advantage of having the fileplan as a taxonomy is that it removes the need to warp the site collection/site/sub-site structure into the form of a fileplan.

It is not easy to enforce a fileplan set up as a SharePoint taxonomy, particularly with the in-place records management option. This is because SharePoint gives teams a lot of options on what type of object to use when collaborating on work. It is hard to predict whether any particular team will want a sub-site, a document library, or a folder for their piece of work. This makes it harder to issue a blanket policy rule such as ‘every document library must be classified against the fileplan in the managed metadata service’

One option might be to use content types, and specifically to use the SharePoint document set content type. In theory it is possible for an organisation to use ‘document set content types’ as a fourth level fileplan classification, with each ‘document set content type’ linked to a specific pathway in the three level fileplan in the managed metadata service. The downside of this approach is that it would force people to use document sets to manage documents of their work. Document sets are a new feature in SharePoint 2010, and at the time of writing I have not seen any case studies from organizations using them in practice.

Conclusion

SharePoint is now a mature collaboration system, having gone through several iterations of the product. But the records management in SharePoint is not yet mature. The records centre in SharePoint 2007 could not be made to work to any scale without extensive custom coding. Many of the concepts on which records management is based in SharePoint (content types, document sets, routing rules, and the records centre) are unique to SharePoint. Records management in SharePoint is still unproven, and will remain unproven until we know of a reasonably sized organisation who have successfully used it for records management across their full range of activities without extensive customisation.

For these reasons I think it is important that we as a records management community look on SharePoint 2010 with a sceptical, questioning eye.

Yesterday I gave a joint presentation in Sheffield on SharePoint in higher education to IWMW10 (an annual workshop for people running websites in UK higher education institutions). I was presenting with Pete Gilbert of the University of the West of England. There is no love for SharePoint amongst web professionals in higher education, which made the reaction to the talk very interesting (and in particular the Twitter backchannel under the hashtag IWMW10) .

Before I talk about the backchannel’s reaction, I’ll describe some conversations I had in the tea break immediately after the talk.

Pressure on web teams to use SharePoint as a content management system
Two people from two different universities came up separately to tell me the same story: they are running a web site, and they have experienced unwelcome pressure from their IT departments to use SharePoint as their web content management system (CMS):

• One of the webmasters is working at an institution which has abandoned a selection process to choose a web content management system. Instead their IT department, in conjunction with the Marketing department, has decided to pick SharePoint as their web CMS, without involving the wide group of internal stakeholders that had been part of the abandoned selection process.
• The other webmaster responded to similar pressure at their institution by dedicating time to finding out about SharePoint and successfully making the case that SharePoint was not the right Web CMS for them

Oxford University’s SharePoint implementation
Ian Senior from Oxford University updated me on their implementation of SharePoint as an internal collaboration system. SharePoint provides a great many configuration choices for owners of team sites. Oxford have sought to reduce this complexity by narrowing the scope of their implementation. They are concentrating on rolling out SharePoint to research teams, and to committees, and have devised template team sites tailored for those particular pieces of work.

Edinburgh Napier’s SharePoint implementation
David Telford from Edinburgh Napier University came up to discuss their SharePoint implementation. Napier are the nearest thing to a SharePoint flagship in UK higher education. They are running SharePoint for their external web site, their internal web site, as a student portal, and to provide collaboration sites for staff. This is supported by a team of four .Net developers. David says that on average two Universities a week come up to Edinburgh to see what they are doing with SharePoint. This creates a pressure on web teams in other institutions, as some of the visitors to Edinburgh Napier return to their own institutions expecting their web teams to replicate the Napier example, using SharePoint.

My position on SharePoint
The Twitter back channel during our talk was unanimously hostile to SharePoint, and gave some very valid criticisms of the product.

I am used to hostility to SharePoint, because my home milieu is the records management community, where most people dislike SharePoint profoundly. However people in the records management community are usually hungry to hear about SharePoint, whether that is because they wish to marshall arguments against it, or to find ways of mitigating the weak records management functionality in it. I got the impression at IWMW10 that there was hostility to there even being a session on SharePoint at the conference, which surprised me.

In the face of such a reaction it is hard not feel a pressure to take a position for or against the product. I have no interest in taking a position for or against SharePoint. My profession is as a trainer and consultant and my job is to help people understand complex phenomena so that they can make decisions about them. I talk about SharePoint because it is there. There are some moral issues in life you have to take a stand on, but I don’t see SharePoint as one of them. I have seen plenty of reasons to be cynical about the SharePoint phenomena, but also reasons to admire some of the things that fellow professionals have done with SharePoint.

Reasons to be cynical about SharePoint
I have seen an IT director bulldozing through SharePoint too quickly and without thought or preparation in order that he could add ‘rolled out SharePoint to an entire organisation’ on his CV. I have seen IT departments undermine established document management systems in organisations by simply signing enterprise agreements with Microsoft and rolling out or making available SharePoint team sites.

I agree with the person I spoke to last year who questioned the ethics of Microsoft promoting SharePoint 2007, which had no support for accessibility standards, for use as a web content management system in higher education.

In my own professional area, records management, Microsoft themselves have behaved cynically. They released SharePoint 2007 with records management functionality that was half-baked and not ready for use. Instead of admitting its inadequacies, they continued to claim that their product could be used for records management purposes. Microsoft contracted an external organisation to write some code that they did not support, and that would prove useful to no-one, purely in order to gain certification for SharePoint 2007 against the main US records management standard.

Prior to the release of SharePoint 2007 most IT departments were not interested in what content management their organisation used, or which collaboration or document management system was chosen. The relevant professionals in those areas (web teams, knowledge managers, records managers) would normally be able to take the lead on that. As soon as Microsoft, with its established relationship with IT departments, issued an enterprise content management system of their own in the form of SharePoint, you began to get IT departments promoting SharePoint. The fact that SharePoint client licences are available to IT departments bundled up with Window, Exchange and Office effectively pulls the rug from under the feet of professionals wanting to chose an alternative system.

There is a legitimate debate about whether or not SharePoint can be regarded as free. You still have to pay for servers, for server licences, for SQL server licences, and if you want to use it for an externally facing web site or extranet you have to pay for an external connectivity licence. And of course you are paying for the bundle itself in the form of the Microsoft Campus Agreement or Enterprise Agreement. But the fact remains that is perceived as free or very cheap by organisations who would be signing that agreement with Microsoft anyway.

Reasons to admire things that people are doing with SharePoint
Despite my cynicism about some of the actions of some IT departments in relation to SharePoint, I also find some uses of SharePoint interesting and even inspiring. I admire implementations like those at Imperial College and of Pete Gilbert at UWE. They both started small with SharePoint, built up knowledge of the product, and offered it as a solution where teams have come to them with a particular need or problem. These implementations have tended to concentrate on SharePoint for internal collaboration, rather than SharePoint as a web site. I also admire David and his team at Edinburgh Napier and their pride in the fact that they have made a tool, that many find difficult to use, fit the needs of their institution.

What I am admiring here is not SharePoint as a tool. It is the professionalism of people like Pete at UWE, David and his colleagues at Edinburgh Napier, and of the team at Imperial. This professionalism manifests itself in their efforts to get to know SharePoint, and in their disinterested and honest advice to colleagues on the circumstances in which SharePoint would be (and would not be) useful to them. That professionalism would be equally admirable if Napier were using Plone as a web content management system, or if Pete at UWE was rolling out Lotus Notes.

That professionalism and care is a million miles away from an IT department choosing SharePoint for the sake of it, which is a practice I deplore.

We are only a decade into the 21st century, but it is already apparent that there will be something different about the records produced by this century, as compared to those of any other time period.  It is not just that they are going to be in electronic rather than hard-copy form, it is more fundamental than that.

Records in previous centuries

The United Kingdom’s National Archives in Kew houses an unbroken eight hundred years worth of records. In every previous one of those 8 centuries of English/British record keeping, there has been a common method of keeping records.  A ubiquitous method that was used for record keeping by any and every organisation.

• In the 13 and 14th centuries the ubiquitous method of record keeping was the roll.   Manorial courts, ecclesiastical courts and the royal administration all kept records in the same way. They would issue a communication (a charter or a judgement for example). Their clerks would copy the communication out onto a strip of parchment and sew it onto the end of the copy of the previous communication, creating a roll
• By the nineteenth century the ubiquitous method of record keeping was the register and the book. Government ministries, branches of the armed forces, companies and trading houses all kept records the same way.  Clerks would keep a register of all incoming and outgoing letters, with one line of description for each letter. Outgoing letters would be copied into a letter book before they were sent out.  Incoming letters would be bound into books, or gathered together in boxes or files.
• In the twentieth century the ubiquitous method of record keeping was the file. Organisations had grown in size, and now performed a great diversity of different types of work.  It was the age of the bureaucracy.  It was no longer possible for a small group of clerks to capture all incoming and outgoing communications within a series of books.   Files were a more flexible record keeping format, a file could be created whenever a piece of work started and closed when it finished

Was there a single organisation, operating in the second half of the twentieth century, that did not create hard copy files to record their work?

Records in our century

In the 21st century there is no such ubiquitous method of record keeping.

Records are still being kept.  But they are all over the place.   When you work with an organisation now you find different teams making different choices about how to record their work, and some teams not making a choice but leaving it down to individuals.   There is a safety net of sorts in the shape of the e-mail inbox, the place of first and last resort to find information in the enterprise.

The range of different ways in which people choose to keep records is dazzling.   I recently visited a compliance team in an organisation.   The team were conscientious, for each area of their work they had made a rational choice of how they keep their records:

• they kept records of their audit work as shared Outlook folders into which they dragged relevant incoming and outgoing e-mails
• they kept records of their policy work, and their approval work, as folders on the shared drive
• they kept records of their inspections on a dedicated line of business system
• they kept records of their project work as folders on the corporate document management system

The organisation concerned, like many others, is intending to roll out SharePoint over the coming year. That will provide yet more choices in terms of where and how teams store the records of their work.  They might use a SharePoint document library for each piece of work.  Or they might set up a SharePoint team site, or sub-site, for each piece of work.

It is significant that SharePoint is likely to come in alongside, rather than instead of, the systems that are already available to the team.

The implications of the lack of a standard method of keeping records for the records management profession

In such a situation we as a records management profession have three options:

• We can try to pursuade the 21st century to adopt a particular application or fomat for record keeping
• We can wait until the 21st century settles on a method of record keeping and in the meantime manage the vast array of different formats as best we can
• We can try to find a way of managing records that allows people to work (and store records) in any application that they choose, whilst at the same time allowing the organisation to control and manage those records over time

Each of these options has its pitfalls

The records management profession will not be able to dictate to the 21st century how it should keep records

The records management/archives profession has never been able, at any point in history, to dictate the way records were kept. In the 20th century records managers did not create the hard copy file.  It was the other way round – the hard copy file created records managers. Organisations were creating hard copy files anyway. Records management came into existence to solve the problem of how large 20th century organisations managed hard copy files at a corporate scale – to help them manage the thousands of files that were being created across a huge variety of different types of work.

In the first years of the 21st century the archives and records management profession tried to create a ubiquitous method of record keeping for the digital age.  It was to have been the ‘record folder’.  Note that we couldn’t take the word ‘file’ with us into the digital age because it had already been commandeered by Microsoft and others to denote a single digital object, such as a document produced in  MS Word.

Electronic records management systems (EDRMS) were specified by various national and pan national government bodies.   Record folders would be the equivalent of hard copy files.  They would be created when a piece of work started and closed when a piece of work finished.  They would be classified within a corporate fileplan.  They would receive the appropriate retention and access rule from the fileplan.  They would protect any document declared to that folder from amendment and from premature deletion.

Records folders have not become ubiquitous.  Some organisations use systems that support them, most do not. The phrase ‘record folder’ has not entered common parlance, it is a piece of jargon that any organisation implementing an electronic records management system has to establish, define, explain and promote.

The 21st century may never settle on a uniform way of keeping records

There is no guarantee that the 21st century will settle on a uniform way of keeping records. The competition between technology companies is driving constant development of new formats for the communication and storage of information. This competition is fuelled by the fact that the world wide web acts as a giant laboratory in which new collaboration/communication formats can be developed, tried and tested. When these formats are successful on the web they are inevitably brought into the enterprise after a time lag.

In the 20th century there was competition between companies supplying organisations with paper, filing stationery and filing cabinets. But none of those companies were trying to disrupt the way individuals communicated or stored information. No company was trying to come up with an alternative to the hard copy document, or an alternative to the practice of gathering hard copy documents together into files. They could not have been able to disrupt these practices even if they had wanted to.

In the 21st century there has been a constant stream of new formats and applications disrupting the way people record and communicate information. Documents are still important. But successive waves of new formats (e-mail, instant message, blogs, discussion boards, wikis, status updates) have come in alongside the document.  

Microsoft still dominate the communication and storage of information in the enterprise. The two dominant business formats are still the document and the e-mail. They are created, stored and managed with Microsoft tools: Word, Outlook and SharePoint. In 2009 Google launched a beta version of Google Wave, an innovative format for collaborating on outputs that explicitly tried to re-magine both e-mail and the document. Google wants to muscle into the enterprise, and would find it easier to do if they could move people onto a different format, a format invented by Google. We do not yet know whether or not Google Wave will be a success, so far it has not lived up to the hype. We do know that whenever a status quo is established there will always be another tech company trying to disrupt it.

This disruption is bound to continue: market dynamics will see to that. Even if the 21st century did settle on one method/application for keeping records, it would be disrupted within a decade.

We have no experience of implementing records systems that manage content stored in various different applications

We can characterise the situation at the end of the first decade of the 21st century as follows:

• Records can be, and are, generated in any application and any format that is generally available to knowledge workers
• More and more different applications and formats are available to 21st century knowledge workers
• More and more different formats and applications will continue to be made available
• The attempt to persuade/ cajole/ force colleagues to save all content needed as records into one application (the electronic records management system) looks doomed to only partial success

Lets make three assumptions

• organisations will continue to exist
• organisations will continue to want to have a corporate method of managing records
• organisations will continue to look to vendors of electronic content management (ECM) systems for solutions to the problem of managing records

If you can’t predict what format or what applications people are going to use within your organisation then your only option is to have a system that will manage records produced in any format, in any application.

The challenge for ECM vendors is :

• Make us a tool that can manage records kept in any format, in any application.  Convince us that it could be used to capture and manage records created in formats and applications that have not been invented yet

The challenge for us as a records management profession is even harder:

• If and when vendors did provide us with such a tool, how would we deploy it?

There is a vendor term for this model: ‘manage-in-place’. Vendors describe these systems as enabling you to hold a fileplan and retention rules within a manage-in-place Enterprise Content Management System (ECM), but apply them to content held elsewhere. For example you could apply a retention rule to content held in shared drives.

The rise of SharePoint has pushed ECM vendors out of the collaboration space, and sent them scrambling to find a new unique selling proposition. The ‘manage in place’ model seems to provide ECM vendors with a way of staying relevant to document and records management within the enterprise. Their new USP of the ECM is – ‘we can help you govern the content that your users create in SharePoint, shared drives, line of business systems etc.’

However there are a great many unanswered questions about the model. Even if the functionality is there, no-one knows how organisations could make it work. There are no case studies out there that I have heard of, and no guidance notes from any national archive or professional society.

Here are some questions about the implementation of a ‘manage-in-place’ records management system:

• Would all colleagues interact with it? Or would it sit in the background, used only by administrators and records managers?
• How would colleagues activate it when they wanted something to be captured as a record? For example lets imagine a team are using a shared drive to keep records of the audits that they carry out. They want those records to be protected and governed by the ‘manage in place’ ECM system. How do they go about that making that happen?
• How would colleagues know something had been captured as a record?
• Do records stay in the original application, or do they move to the records management tool?

The main question records managers ask me about SharePoint is ‘do I need to plug an ECM system into the back of it to manage records?’. Explaining the weaknesses in records management in SharePoint 2007 and 2010 is not enough to persuade SharePoint customers to buy ECM systems to manage records. The price tag of ECM licences, and the added complexity of implementing an additional application are strong barriers to overcome. The attitude of people seems to be ‘we are getting SharePoint anyway, we will see how we go with it, and if we need to plug an ECM in to govern the content we will do so at a later date’.

If the ECM vendors seriously want to win market share for their ‘manage-in-place’ records systems then they need to get serious about promoting the concept, explaining the model and sparking a debate about it. They would also need to slash their prices so they get some take-up, some case studies, and some momentum.

In a recent episode of the MOSS Show podcast Hilton Giesenow interviewed two highly experienced SharePoint administrators, Todd Klindt and Shane Young. Todd and Shane described the options for upgrading from SharePoint 2007 to 2010, and why they think the upgrade will be a boon for hardware manufacturers.

If you are running SharePoint 2007 you will only have two options for upgrading to SharePoint 2010:

• An in-place upgrade: SharePoint goes through all your content databases and converts them to SharePoint 2010. It does it all in one go, on the same hardware, with everything staying at the same url address as previously
• A database attach upgrade: you keep your SharePoint 2007 farm running, and set up a brand new server farm to run SharePoint 2010 alongside it. You move your content databases over one by one from the SharePoint 2007 farm to the SharePoint 2010 farm

At first sight the in-place upgrade appears the better option. It is quicker, simpler (because you don’t have the complexity of running 2007 and 2010 in parallel) and cheaper (because you can re-use your existing hardware).

However Todd and Shane said that their experiences of running in-place upgrades from SharePoint 2003 to 2007 were so bad that they are wary of trusting it this time around. The in-place option gives you no control over the upgrade process. You have to let the whole upgrade take place over the whole farm. If the upgrade has failed all you can do is revert completely back to SharePoint 2007, diagnose what caused the problem, fix that problem and try the whole thing all over again.

Todd’s experience of running in-place upgrades from SharePoint 2003 to 2007 was that diagnosing the problem that caused your in-place upgrade to fail was like ‘shooting fish in a barrel’ . The problem could have come from anywhere on any content database anywhere on the farm, and the error messages that SharePoint provided were of little or no help.

Todd and Shane will instead be recommending that their clients run a database attach upgrade. They admit that this is a more expensive, time consuming and complex option than the in-place upgrade. It requires you to buy a complete set of new hardware for SharePoint 2010 to run on. It results in have what Todd called ‘URL craziness” as you move the content databases over one by one to run on the SharePoint 2010 farm, whilst still running the remaining databases on SharePoint 2007 from the 2007 farm.

However the advantage of the database attach upgrade is that you have more control over the whole process. You isolate each content database and move them over one by one. If the upgrade to one database fails it doesn’t mean that you have to roll back the whole of the rest of the upgrade. And it is much easier to diagnose problems because you know which content database the problem has occurred in.

Todd and Shane said that if they were a hardware manufacturer like HP or Dell they would be rubbing their hands at the thought of all those upgrades from SharePoint 2007 to 2010, and all the extra hardware that organisations will be buying.

Organisations today are using two conflicting working definitions of a record. Neither definition is adequate on its own (which is why the two conflicting definitions co-exist with each other!).

The first of the two working definitions states that ‘records’ are those documents/communications that are needed as evidence of a piece of work. This definition comes from records management theory. Records management practice aims to capture these documents in a file/ records folder dedicated to each particular piece of work. The file/ records folder exists in a records system, which could be anything from a shared network drive, to a SharePoint team site, an electronic records management system, a line of business document management system, or a traditional filing cabinet.

Traditionally records management practice aims to ensure that the file/ records folder is the only information about a piece of work that survives over time. This translates into advice to colleagues on handling documents and e-mails such as ‘if it is important put it on the file, it if it isn’t important get rid of it ‘

If a team is confident that the file/records folder tells the whole story of the piece of work, then they can be confident in disposing of all information about that work lying outside of the file. The converse is also true – if you have not got confidence that the files in your records system(s) serves as a sufficient story of your organisation’s activities, then you are not going to be confident that you can delete material from other information stores such as e-mail in-boxes, and shared drives.

However for most pieces of work in an organisation ‘the file’ in a designated record system can no longer be regarded as the single source of evidence on that piece of work. This is down to two factors:

• it is highly likely that other pieces of evidence about that piece of work persist outside of the file, and that these pieces of evidence are findable or discoverable
• it is highly unlikely that each colleague has placed all significant documents/communications arising from that piece of work onto the file

I am struggling to think of a single organisation I have worked with that routinely delete e-mails after a period of months, or routinely deletes material on shared drives that is a year or two years old – they simply have not got enough confidence in their ‘records’ to risk losing important material like that.

The second working definition defines ‘records’ as being any information, in whatever form, that is held by that organisation. In other words any piece of information that exists anywhere: on the organisations network, in its filing cabinets, on people’s desktops, in desk drawers. This definition stems from the legal obligations placed on organisations. These obligations vary from jurisdiction to jurisdiction, but include data protection, freedom of information, and e-discovery. Such obligations make no distinction between those documents/communicaions that an organisation intended to keep as records, and those that have accidentally survived.

This definition is not adequate from a practical point of view either. When an organisation receives a data protection subject access request, or a freedom of information request or an e-discovery demand, they have to pull out all the stops and trawl their network. They might ask swathes of the organisation to go through their e-mail and unearth any messages regarding that matter. They might do searches of shared drives and document management systems. They unearth a pile or a mountain of material, that they then spend days or weeks sifting through. That is a massive pain, that an organisation is willing to put up with because that is the law that they are working under. But this approach would be overkill for most internal uses of record, such as the following scenarios

• You have taken over the management of a contract that has run into problems. You want to understand what has been agreed and discussed with the contractor up to this point.
• You have been asked to revise the service level agreement you offer to your customers. You want to understand the thinking behind the clauses in the current agreement. You want to read the discussions/correspondence that took place the last time the SLA was drafted
• A budgeted cost has changed between different versions of a business plan and you are trying to understand why

In each of the above scenarios you would be hoping that the colleagues involved in those pieces of work had kept a good file/records folder of that work, in which they had brought together all the relevant documents/communications relating to the work.

The dilemma for organisations is that they are caught between a rock and a hard place. They are not able to maintain good files across all areas of work. The concept of a file/records folder feels like an old-fashioned concept, a legacy from the hard-copy world of decades gone by. And yet organisations still need some means of relating together the records (however defined!) of a piece of work and managing them over time.

If the file/records folder was invented today, from scratch, with no history or legacy, what features would it have?

Science Fiction writer Bruce Sterling has posted a transcript of his February 6 talk to Transmediale 10 Atemporality for the creative artist. The talk investigates the impact of the real time web on all those cultural activities that depend in some way on narrative – including the writing of history.

”History books are ink on paper. They are linear narratives with beginning and ends. They are stories created from archival documents and from other books. Network culture is not really into that. Network culture differs from literary culture in a great many ways. And step one is that the operating system is an unquestioned given. The first thing you do is go to the operating system, without even thinking of it as a conscious choice.

Then there is the colossally huge, searchable, public domain, which is now at your fingertips. There are methods to track where the eyeballs of the users are going. There are intellectual property problems in revenue, which interferes with scholarship as much as it aids it. There is a practice of ‘ragpicking’ with digital material – of loops, tracks, sampling. There are search engines, which are becoming major intellectual and public political actors. There is ‘collective intelligence’. Or, if you don’t want to dignify it with that term, you can just call it ‘internet meme ooze’. But its all over the place, just termite mounds of poorly organized and extremely potent knowledge, quantifiable, interchangeable data with newly networked relations. We cannot get rid of this stuff. It is our new burden, it is there as a fact on the ground, it is a fait accompli.

There are new asynchronous communication forms that are globalized and offshored, and there is the loss of a canon and a record. There is no single authoritative voice of history. Instead we get wildly empowered cranks, lunatics, and every kind of long-tail intellectual market appearing in network culture. Everything from brilliant insight to scurillous rumor.

This really changes the narrative, and the organized presentations of history in a way that history cannot recover from.”

What are the implications of this for record keeping? If the writing of history as a single narrative becomes impossible, then is the attempt to keep a single narrative of a piece of work impossible and/or unnecessary too? Should we still try to create a file (electronic or paper) for every piece of work, that brings together all the significant documents and communications arising from that work?

In the same talk Sterling describes a new way of looking at history:

history is a story. And to write down the story of the fourteenth century, to just ask yourself – “what happened in the fourteenth century?” — is a very different matter from asking the atemporal question: “What does Google do when I input the search term ‘fourteenth century?

This is reminiscent of the switch we are witnessing in record keeping. A decade ago organisations were trying to keep good records of everything they did, to be able to tell the story of everything they did, to keep an organised narrative. Now the aim is reduced to being able to respond to e-discovery/ Freedom of Infomation/ Data Protection Subject Access requests.

Our traditional concept of a record is of a stable information resource: a file, sitting in a record store or archive, or in an electronic records management system. A file that stays unaltered and inviolate as it moves through time, a narrative waiting to be read and interpreted by different waves of colleagues, auditors and historians.

The e-discovery/freedom of information/subject access request is to ‘show us everything your systems know about the dismissal of employee y’ or ‘show us everything your systems know about the contract with company x’. The narrative only comes together when the request comes in. The answer to the question will change over time, as the organisation’s network and systems change, as the tools it has to search them change, as things drop off the network and are added to it. In the same way as Google’s answer to a search query on ‘the fourteenth century’ will change over time, as its algorithms changes, as the internet changes.

Even if an organisation has been able to keep a good file for a piece of work the e-discovery/freedom of information/subject access requestor will want to go beyond that file. It will want the organisation to dredge up material kept outside of that file that the requestor might find useful, advantageous or interesting.

The implication for records management is that we are in a transition away from managing static files towards managing shifting networks of information.

Alan Pelz-Sharpe is an incisive Enterprise Content Management system analyst. He blogged today that Google’s aggressive use of its users contact data in the recent launch of Google Buzz indicates that Google is unsuitable for the enterprise, as they can not be trusted with enterprise data.

Alan’s statement is important in the context of the battle between Microsoft and Google for the provision of cloud based e-mail, calendaring and collaboration applications to the enterprise. The battle has only recent commenced but it is already developed into both a price war and a war of words.

• Microsoft’s offering is BPOS (Business Productivity Online Standard Suite). BPOS provides enterprises with the following applications, hosted in the cloud: Exchange Online, SharePoint Online, Office Live Meeting, and Office Communications Online
• Google’s offering is Google Apps which packages up Gmail for business, Google sites, Google Groups, Google Docs, Google Calendar and Google Video

Both of these products are immature: Google Apps has been on the market longer, but Google are new to enterprise computing. Microsoft have vast experience of enterprise computing but little experience of providing software-as-a-service (SaaS).

In terms of functionality BPOS, and in particular SharePoint, provides far more features than Google Apps. If levels of service and price were comparable you would expect Microsoft to win this war (though small enterprises may prefer the relative simplicity of Google Apps).

The fact that Alan Pelz-Sharpe sees trustworthiness as a key factor in judging a SaaS vendor shows how different the SaaS space is from the traditional packaged software space. In the packaged software space you did not have to decide whether you could trust the vendor with your data – they would never see it. You simply needed to be satisfied with the features that their software possessed, the financial stability of the vendor and their road map for their product.

In contrast SaaS providers will be judged on the supplier’s reliability, quality of service, security and trustworthiness, as much as on the features of the software. The SaaS market plays in many ways to Google’s strengths. Their success has been built on the resilience of their infrastructure. Their search engine is hardly ever down. They have always guarded the secrets of how they configure their data centres as zealously as their search algorithm, indicating that they regard running data centres as a core strategic competency.

Conversely the move to cloud computing negates two of Microsoft’s strengths:

• Their monopoly, through Windows, of operating systems in the enterprise is negated because organisations can run applications off rival operating systems based in the cloud, or subscribe to applications (like Google Apps) that run off rival operating systems (like Linux).
• The expertise that enterprise IT departments have built up over the years in the administration, upgrade and extension of Microsoft products gave them an edge against competitors. Organisations were reluctant to buy software that their IT staff were not familiar with. In the SaaS world applications are administered and upgraded by the provider, and the scope for developing and extending them is limited

Google and Microsoft are big, cash-rich companies, with plenty of freedom to manoeuvre, who act in what they see as their best commercial interests. I do not want to condone Google’s aggressive use of Gmail users’ contacts data, (it made me glad I do not use Google to maintain my contacts). However the commercial imperatives behind Google’s over aggressive launch of Buzz are not the same as Google’s drivers to enter the SaaS space, and I do not think we can necessarily assume that Google is less trustworthy with enterprise data than its competitors purely on the basis of their actions in the launch of Buzz.

The reasons behind Google’s aggressive launch of Buzz lie in the threat that Facebook poses to its monopoly of the web advertising market. Google’s monopoly of web advertising stems from their monopoly of search, and from the fact that search was until recently overwhelmingly the major route through which individuals were directed to websites.

The growth of Twitter and in particular of Facebook has altered that. More and more traffic on the web is directed by links from friends on Facebook or Twitter, and this is reducing the influence of Google search results. Steve Rubel recently reported that Facebook now drives more traffic to major websites than Google. In another post Rubel said that ‘Facebook is unstoppable, they aren’t just the next Google, they’re the next web’

Google recognises that in the medium term search will lose out to discovery through social networks. People are more and more using their Twitter network, and/or their Facebook network to bring them news, rather than going out and searching for it on Google. Google will only be able to retain its dominance of advertising if it can provide a discovery service that is both personalised to you as an individual, and social. It needs to be tailored to who you are, where you are, who you are interested in, and who your friends are. This is why Google is turning itself from a search company into a software company, it needs you to use its applications because that is how it will get to know you.

Contacts data was at the crux of the controversy over the Buzz launch, and contacts data is crucial to Google. The key advantage that Google’s Android phones have over Apple’s iPhone is that when you log on to your Google account on a Google phone it instantly synchronises your Google contacts and Google calendar to your phone. In contrast getting my contacts and calendar synched between my Mac and my iPhone is a service that Apple charges me £59 a year for (it is worth it to me, and I love the service, but I do not think the charge is sustainable in the face of Google providing a similar service for free).

Contacts data is important to people. They want to keep it in their control as they move from job to job, and device to device. Facebook and LinkedIn function in many ways as address books, but they do not tend to hold phone numbers and have no coverage of people that are not users of those applications. The perfect contacts book would be available when I use my phone and when I use my laptop. It would link with my e-mail, and it would have links to my contacts’ identities on applications such as Twitter, Facebook and LinkedIn.

Google are in a better place than their rivals to provide just such an integrated contact service:

• Apple are dominant in smartphones, and the iPad could give them dominance of the netbook/tablet space. Their contacts and calendar applications are more pleasant to use than Google’s. But they are weak in e-mail provision and do not provide a social network.
• Google have the smartphones, the e-mail service, and now with Buzz they are making a play for the social network space. When the Chrome OS netbook is launched they will rival Apple in the netbook/tablet space
• Facebook is by far the strongest in the social network space, but does not (yet) do e-mails or phones or any sort of device. There have been rumours about a Facebook phone and a Facebook e-mail service for some time and both would make sense.

Google’s land grab with Buzz was a bid to get into the social network space before Facebook’s extension outwards became unstoppable. They needed to get as many people as possible using Buzz as quickly as possible and that is why they launched it on users with little or no warning. They pushed Gmail users at breakneck speed through a series of questions without giving them any way of judging what the consequences of their choices were. Google used Gmail users’ contacts to suggest people for them to follow, and to suggest them to others. However the consequence of this aggressive strategy was that some people found that lists of people that they frequently had Gmail correspondence with were being displayed on their Google profile.

It was an aggressive move, but it is an aggressive market. Google has more to lose from standing still and seeing Facebook overtake it than it does by aggressively moving to occupy some of Facebook’s space.

Alan Pelz-Sharpe points out in his post that the enterprise market accounts for only 1% of Google’s revenue. He is right that the enterprise market is far less important to Google than it is to its competitors (for example Microsoft). However Google’s war with Facebook is not taking place in the enterprise. Facebook is not trying to compete for the enterprise (Microsoft holds a significant stake in Facebook and the two companies have not encroached on each other’s space). I do not believe that Google has anything to gain by acting as fast and loose with enterprise data as it did with customer contact data when it launched Buzz. For these reasons I am reluctant to read across from their actions in the launch of Buzz to make any conclusions as to the trustworthiness or otherwise of Google in relation to enterprise data

Paul Buchheit is the brains behind Gmail. He then left Google and founded Friend feed which was recently acquired by Facebook.

Last week Paul blogged his opinion on the design of devices and applications.

Buchheit’s post was written in response to criticisms of Apple’s forthcoming iPad. Critics have listed all the things that the iPad won’t do that they would expect a tablet computer or netbook to be able to do. For Buchheit these critics are missing the point:

What’s the right approach to new products? Pick three key attributes or features, get those things very, very right, and then forget about everything else. Those three attributes define the fundamental essence and value of the product — the rest is noise. For example, the original iPod was: 1) small enough to fit in your pocket, 2) had enough storage to hold many hours of music and 3) easy to sync with your Mac (most hardware companies can’t make software, so I bet the others got this wrong). That’s it — no wireless, no ability to edit playlists on the device, no support for Ogg — nothing but the essentials, well executed.

We took a similar approach when launching Gmail. It was fast, stored all of your email (back when 4MB quotas were the norm), and had an innovative interface based on conversations and search. The secondary and tertiary features were minimal or absent. There was no “rich text” composer. The original address book was implemented in two days and did almost nothing (the engineer doing the work originally wanted to spend five days on it, but I talked him down to two since I never use that feature anyway). Of course those other features can be added or improved later on (and Gmail has certainly improved a lot since launch), but if the basic product isn’t compelling, adding more features won’t save it.

By focusing on only a few core features in the first version, you are forced to find the true essence and value of the product. If your product needs “everything” in order to be good, then it’s probably not very innovative (though it might be a nice upgrade to an existing product). Put another way, if your product is great, it doesn’t need to be good.

At the end of the post Buchheit made a comment that helps explain why so many enterprise applications suffer useability problems:

Disclaimer: This advice probably only applies to consumer products (ones where the purchaser is also the user ..). For markets that have purchasing processes with long lists of feature requirements, you should probably just crank out as many features as possible and not waste time on simplicity or usability.

Enterprises still purchase their applications and devices through a long list of feature requirements. Microsoft are kings of enterprise computing because they provide so many features in their products- SharePoint can be made to do almost anything. It isn’t just Microsoft that operates like this. Every Enterprise Content Management system vendor plays the same game, favouring features over useablity and design values, because it is features that they are judged on.

This contrast between the design of devices and applications intended to appeal directly to consumers and those intended to win through enterprise procurement processes is coming to a head.   More and more aspects of enterprise computing are becoming markets which consumer-targeted devices/applications are competing for.

Three years ago the battle for the smartphone market was a straight head to head between Research in Motion’s Blackberry and Microsoft’s Windows Mobile. Both assumed the smartphone market would go to the supplier that won the enterprise market,  and both crammed their smartphone full of features. Then Apple bought out the iPhone, with less functionality,  appealing direct to consumers, and a design that so intuitive that they did not have to ship a manual with the phone.  Now the future of the smartphone is a battle between two consumer companies: Google and Apple.

These two companies will also be the ones fighting it out for the tablet/netbook market. This year will see the launch of Apple’s iPad, and of Google’s Chrome OS – a new operating system for netbooks. Both will be bought as consumer devices but used for work. Read this post from Pete Gilbert for a discussion of how the iPad will support work in ‘the third place’ – outside of both the home and the office, on the move, in cafes, on trains, airport lounges etc.

The trend of enterprise computing is to require less and less to be installed on the PC, with the logical end result being that your device will only need a browser. For over a decade Microsoft have had a very safe income stream from Microsoft Office, which always had to be installed on the client device. Now it has to compete with Google Docs, which not only doesn’t require anything installed on the device, but is also free. Their solution is Office Web Apps, which will allow people to view and edit documents held in SharePoint 2010′s without having MS Office installed on the device.

As soon as you reach the point where only a browser is required on the device then (in theory!) it doesn’t matter what device an employee uses (so long as it doesn’t compromise security). That will work to the advantage of both the enterprise and the individual: the enterprise because they will no longer need to worry about providing every individual with a PC (and configuring and maintaining them) – the individual because they can choose their device of preference.

Consumerisation is a trend that affects applications as much as devices.

The one thing that every web 2.0 application has in common is that all of them were written to be used without training and without helpdesk support. It is conceivable that enterprises may start to regard training and helpdesk support as unwanted costs of applications (much like cloud computing vendors are encouraging enterprises to see system and server administration as unwanted costs).

Last week I heard Roger James, Director of IT at the University of Westminster, describe to the UNICOM Records Management update conference how the University provided Google Apps to all its students and staff without having to provide any training. It has been in use for over a year and they have so far only received 150 help desk calls. Google Apps is an amalgam of applications such as GMail, Google calendar and Google Docs that were first launched as free consumer products on the web.

This Wednesday I heard Simon Wardley give a fascinating overview of cloud computing to the Records Management Update 2010 Conference organised by UNICOM. Simon is responsible for Cloud Computing Strategy for Canonical.

Simon told us that cloud computing is difficult to define because it is not a thing, it is a transition.

The transition is from:

• computing-as-a-product (the enterprise buys in hardware, operating systems and applications as products which they install, configure, deploy and maintain themselves)

towards:

• computing as a service (the enterprise subscribes to applications which are hosted in the cloud, deployed on operating systems that are hosted in the cloud, sitting on hardware that exists somewhere on the cloud)

The transition period will be problematic and disruptive for companies. They won’t know who to trust, or what standards they should be insisting on, or which suppliers will be viable over time, or how they should govern their information in the cloud. There will exist many different cloud models during the transition period. Enterprises won’t just flick a switch and move all their IT infrastructure, all their development and all their applications straight over to the cloud.

Cloud computing is not a new phenomenon. John McCarthy and Douglas Parkhill both (separately) came up with the concept of utility computing in the 1960s. The necessary technology (virtualisation, which allows a provider to run several different virtual servers on one peice of hardware) was described by Jerry Popek et al in the 1970s. But the crucial attitude change came more recently, with the writings of Paul Strassman and Nicholas Carr who argue that the majority of IT activities in an enterprise do not give the organisation any strategic advantage. In other words IT has become a commodity, well enough defined that no enterprise gets a strategic advantage from doing it better or differently than their competitors.

Simon told us that some organisations will lose out in the transition. One of the key risks is supplier lock in. To avoid supplier lock in you need to be able to get your data out of your existing supplier. You also need alternative suppliers to be both willing and able to accept your data in the form that you extracted it from the previous supplier. The cloud has not yet reached that level of maturity, it is still in a transition period and there is no agreed de facto standard for APIs (the application programming interfaces through which organisations access, amend and add to the data held by their cloud provider). Most cloud providers have set up their own APIs to their services. Simon showed us a long list of different cloud APIs. If you choose a supplier whose API is not compatible with any other vendor then changing supplier would be hampered by the cost of adapting their data so that it can be deposited through the different API.

Simon showed us the complexity of the cloud computing stack. with its three levels of infrastructure, platform and applications.

• Software-as-a-service vendors provide applications – tools such as Google Apps, SharePoint Online and Salesforce that end users can utilise
• Platform-as-a-service vendors provide programming language support and development tools to enable an enterprise to develop new applications and to deploy those applications [see this information week article for a survey of the platform-as-a-service market]
• Infrastructure-as-a-service vendors provide the’ bare metal’ hardware, plus an operating system and a virtualisation tool

Simopn anticipates that there will be cloud computing disasters and failures.
Cloud computing creates dependencies up and down this stack. For example you may trust your software-as-a-service provider to provide you with a stable, reliable application. They in turn may trust their platform-as-a-service provider. But if the infrastructure-as-a-Service provider at the bottom of the stack fails, this would have a knock-on effect up the stack causing both the platform-as-a-service and the software-as-a-service to fail. Simon drew parallels with the the catastrophic banking collapse of late 2008, which resulted from bankers being shielded from the true risks of their investment through the existence of different layers separating their investments from the properties on which those investments were secured.

However Simon does not anticipate that the risks, disruption and losses involved in the transition to cloud computing will prevent that transition happening. Organisations that continue to host their own IT may find themselves at the wrong end of a competitive gap between them and organisations that use cheaper, more efficient cloud computing services. This may mean that many organisations will have eventually no choice but to move to a cloud computing model, in some shape or form, for most non-strategic computing provision.

The noughties was a decade of two halves for electronic records management systems in the UK

The first half of the decade: 2000 to 2005

The first half of the decade was a boom period. The UK National Archives (then called the Public Record Office) issued an influential statement of requrements (TNA 2002) for electronic records management systems, with an accompanying evaluation programme. Similar evaluation programmes were set up in other countries. The requirements included the ability to:

• hold a hierarchical corporate records classification (fileplan)
• link retention rules and access rules to the fileplan
• allow folders and documents to inherit retention and access rules from the heading of the fileplan that they are saved under

The acronym EDRMS (electronic document and records management systems) was applied to systems that complied with statements of requirements such as TNA 2002. In the UK large organisations in the public sector and in regulated sectors of the economy would routinely specify TNA 2002 compliance when issuing a tender for a corporate scale document management system.

Like most booms it was overdone. Some UK central government departments rushed into implemementations, spurred on by a Modernising Government target (all UK government departments must create new records electronically by 2004), and by a desire to be ready for the coming into force in January 2005 of the Freedom of Information regimes for England and Wales, and for Scotland. Corporate Fileplans proved difficult for organisations to construct and unpopular for users to use.

Nevertheless EDRMS had established itself as an integral part of the enterprise content management (ECM) landscape. All the big ECM suite vendors (IBM, Open Text, ECM, Oracle etc.) now provide an EDRMS offering within their suites, alongside their web content management, portal and workflow offerings.

The second half of the decade – 2005 to 2009

So here we are at the start of 2010. The acronym EDRMS has almost disappeared from our language. What are the factors that will shape the way that the electronic records management system market develops over the coming years?

Make or break for MoReq2

MoReq 2 came out at a bad time for records management standards and evaluation programmes. The double whammy of the global downturn and the rise in SharePoint meant that there was less money to be made out of electronic records management systems. In order to comply with MoReq2 all vendors, even those meeting existing standards such as TNA 2002, would need to make investments in their products to make their necessary changes. Most of the vendors decided not to make those investments and have not yet submitted their products for testing. The result is that at the time of writing only one vendor (Fabasoft) has acheived compliance with the standard. In November 2009 I heard Doug Miles of AIIM tell an AIIM trade members meeting that so long as there was only vendor on the list then buyers would be unlikely to specify MoReq 2 compliance on their Invitations to Tender, because if they did so they would end up with a shortlist of one supplier.

As a result of this situation MoReq2 will be overhauled during the early part of 2010, with a view to making it less onerous for vendors to adapt their products. Some functionality that is currently labelled as mandatory will be downgraded to optional.

Rory Staunton sits on the MoReq2 governance committee, and he has written an interesting article in the January 2010 Records Management Bulletin. He does not pull any punches in the article.
He says:

The market reality is that records management software as a component market has almost collapsed. The sales of records management software licences from vendors have been falling before and during this recession, and most vendors are chasing a non-existent goal they call eDiscovery. US style eDiscovery is almost non existent in Europe, and MoReq2 in its original version was a basket case

Electronic records management has a tough battle ahead of it. Staunton believes that the best chance of MoReq 2 having an influence is to pursuade regulators to adopt MoReq 2

The way to win a game, of course, is to write the rules, so clever national archives, vendorrs and large users should target regulators in all sectors and propose that they adopt and extend MoReq2 based approaches…..The real success of MoReq2 will be judged in five years time, depending on the number of regulators that embracee and extend it.’

Staunton is thinking or regulators in fields ‘from envirnoment to nuclear, healthcare to telcos, retail to finance’

Staunton went on to say:

‘there is not time to waste if the records management community are to ride the wave of impending business regulations that are sure to emerge from the current and widespread breakdown in financial sector regulation. In market terms, Europe needs some basic approach to securing business information. Europe needs a more productive approach than the knee-jerk, lawyer-led, e-mail centric eDiscovery systems that the North Americans are adopting, which are a retrospective friction on business, and do not enhance compliance as they cure, rather than prevent, breaches of process’

The coming of SharePoint 2010

John Newton (chairman and CTO of Alfresco, a competitor to SharePoint) paints a bleak picture of the effect of SharePoint on ECM vendors, in a blogpost outlining his predictions for 2010:

You have to hand it to Microsoft; they seem to have scared the bejesus out of the traditional ECM vendors. They have all seemed to rolled over and played dead in the wake of SharePoint 2007

And it could get worse before it gets better. SharePoint 2010 is coming along with ‘ ever tighter integration with office, better web support, new records management, and claims to better scalability and administration’

Nevertheless Newton believes that SharePoint still has vulnerabilities, arising from the fact that organisations adopting it are tied into the whole Microsoft stack :

”Microsoft has chosen not to address its fundamental architectural flaw of storing content in the database and it is still an exclusively Microsoft-centric platform. Forget whatever database, operating system, language, browser you have – you better get used to SQL-Server, .NET, Windows, IE and don’t forget Silverlight. If the traditional vendors can’t battle the crap out of that, then they deserve to lose. The next 12 months will be critical during the transition to SharePoint 2010”

ECM and SharePoint: competition or partnership?

The reaction of ECM vendors to SharePoint 2007 has gone through 3 phases:

• At first ECM vendors believed that the difficulties of governing information in SharePoint 2007 would mean that SharePoint would not greatly threaten their market share.
• When it became clear that SharePoint 2007 was shipping at a rapid rate the ECM vendors started developing ‘web parts’ that could be deployed within the SharePoint environment. The web parts enabled people to save docutments to the ECM, edit them and search for them, without leaving their SharePoint environment.
• More recently vendors have realised that a much deeper integration with SharePoint is required. The problem with using web parts for integration with an ECM was that SharePoint knew nothing of the documents. Clients wanted the content to be available to SharePoint so that they had the option of using SharePoint functionality (such as workflows) with the documents and other content that they created.

Most of the ECM vendors that I have spoken to recently are adopting an approach to integration with SharePoint, 2007 which sees content stored in the ECM system, but made available to SharePoint so that to all intents and purposes SharePoint views it as SharePoint content

The ECM vendors have a reasonable tale to tell. Their selling proposition is:

• that you can use the ECM to house your fileplan, and to apply it to any system that you connect it to (not just SharePoint)
• you can apply the fileplan to any type of content (not just documents, but also content such as discussion board posts, blog posts etc.), and any level of granularity (not just documents, but folders of documents, document libraries or entire team sites)
• that they provides a more efficient model for storing content. SharePoint stores both the document/content itself and the metadata about it, in a database (SQL Server). ECM systems store the document/content in a repository, with only the metadata stored in a database
• that although the content is stored outside of SharePoint/SQL server, the content is made available to SharePoint so whatever you wanted to do in SharePoint with that content you would still be able to do.

However the vendors are facing an uphill battle. The problems for them are that:

• Most of the ECM vendors seem to have settled on the same model. They are fighting for a relatively small market, and customers may find it hard to differentiate them from each other.
• they are reacting to SharePoint, rather than causing problems for Microsoft. It is one way traffic. There is nothing that any ECM vendor has done over the past few years that Microsoft has felt compelled to react to. Microsoft has nothing to fear from the ECM vendors in the collaboration space. (Note that things are slightly different in the web content management and portal spaces where SharePoint has had less impact on the market share of traditional ECM vendors).
• As Microsoft gradually improve information governance within SharePoint they will leave less and less space for ECM systems in the collaboration space.
• ECM vendors will be reluctant to rock the boat with Microsoft. Several ECM vendors have told me that Microsoft has been very helpful to them, sharing their roadmap and plans for 2010 well in advance so that the ECM vendors can plan how their products will integrate with it. But it means that in the collaboration space ECM vendors are becoming partners rather than competitors with Microsoft.

It is interesting to note that the only two vendors that are trying to take Microsoft on in the collaboration space (Alfresco and Google) are not traditional ECM vendors. Alfresco provides open source collaborative and document management software. Google provides Google Apps from the cloud, as a subscription service.