On 5 April this year I sat on a panel at the UK Information and Records Management Society (IRMS) conference. We were each asked for our view of the future of technology. We talked about the cloud, and also of the end of the dominance of ‘the document’ as a format. One member of the audience brought us back down to earth with the comment ‘never mind the future, we don’t know how to manage records in all the many and various systems our organisations have now’.

Ed Fowler of Cap Gemini said that organisations will need to get used to the fact that their records will be created and stored in many different content repositories, and that few if any of these repositories will have records management functionality.  Ed said records managers should abandon any hope that they could specify and implement one corporate electronic document and records management system (EDRMS) and expect all their colleagues to save their records into it. Recently he had only seen one UK organisation tendering for an EDRMS.

Ed was announcing the death of the traditional EDRMS. I would add a caveat here – information systems don’t die, they just lose momentum. EDRMS has lost momentum but most of those organisations that have succeeded in implementing them corporate-wide have felt a benefit and will continue to maintain and develop their systems.

The loss of momentum in EDRM is nowhere better illustrated than with MoReq, the European Union’s specification of electronic records management (ERM) systems. In early summer 2008 MoReq 2 was launched, only to be scuppered by two tornadoes: the global economic downturn, and the rise of Microsoft’s SharePoint (which took the collaboration space out of the reach of EDRMS vendors). By the end of the following year, 2009, only one vendor had submitted a system for testing against the specification, and the decision was taken by its sponsors, the DLM Forum, to commission a radical rewrite to the MoReq specification. The rewritten specification, called MoReq 2010, will be launched at the next meeting of the DLM Forum, in Budapest, next week.

The previous day (April 4) of the IRMS conference had seen a debate between Jon Garde (author of the new MoReq 2010), and Marc Fresko (author of its predecessor, MoReq 2).

Jon Garde introduced MoReq 2010 by arguing that we needed to revisualise the ERM system.  Previous specifications saw an ERM systems as a stand alone content repository that stood alongside the other content repositories within the organisation. MoReq 2010 sees ERM as a capability that could be embodied within each separate application that the organisation uses, or that could sit behind those applications and manage records created within them.

Marc Fresko was critical of MoReq 2010. In his view it introduced too many new concepts too quickly and the timeframe for the publication of the specification has left too little time for debate.

The fact that Jon and Marc differ so sharply in their viewpoints shows how different MoReq 2010 is from previous versions of MoReq.

Previous versions of MoReq aimed to specify systems that could serve all the records management needs of any organisation, in any sector. They created the breed of information system called the EDRMS. The idea was that an EDRM system would be rolled out to all users, and would hold within it a hierarchical classification (called a business classification or fileplan) that covered all the work of the organisation. Users would capture and store documents and e-mails needed as records into the EDRMS, within files (folders) classified against the business classification/fileplan.

The traditional EDRMS could not on its own solve the problem of multiple repositories. In organisations some functions (types of work) are more important than others. The more important functions will usually have a line of business application procured or developed specifically for them. An EDRMS, because it aims to cover all the diverse activities of an organisation, by definition cannot be tailored to any one type of work.  This means even those organisations with corporate-wide EDRM systems still get the problem of managing records in multiple repositories. Once SharePoint came along this problem was made even worse – if an organisation is putting a lot of time and effort implementing SharePoint for collaboration they will probably not have the capacity to implement a corporate EDRMS in tandem with it, however much they recognise the problems of SharePoint sprawl and the weaknesses of SharePoint’s own records management model.

MoReq 2010 has been written to encourage different models of records management system to emerge. It does this by adopting a modular structure. All MoReq 2010 compliant systems have to comply with a core set of requirements. Beyond that there are different modules which vendors can chose whether or not to submit their product for testing against. Some vendors may well decide to submit traditional EDRM-style systems for certification. But they are also able to submit systems for testing that meet a completely different model, for example:

• Systems that end-users do not interact with directly, but instead capture and store records that users had created in other systems
• Systems that do not store records, but instead govern and protect records held in other systems
• Line of business systems or single purpose applications that are not intended to be a general records system but which have have the ability to manage the records that they capture
• Systems that can fulfill two or more of those roles – for example a system that could be deployed as a traditional EDRM that some end-users would interact with directly, but which also possessed the capability to manage records held in other content repositories

The old aspirations that MoReq 2010 abandons

MoReq 2010 is a major change from all previous specifications of electronic records management systems. It is a break not just with the previous two versions of MoReq issued by the European Union (MoReq and MoReq 2) but also with the UK’s TNA 2002 standard and the US DoD 5015.2 standard.

MoReq 2010 abandons three big aspirations that all those previous specifications shared. It abandons:

• the idea of the ‘file’ in the electronic records management system being the ERM system equivalent of the traditional hard copy file MoReq 2010 replaces the concept of the ‘file’ with the new concept of an ‘aggregation’ This is a major break with the vocabulary of the hard copy era. In a MoReq 2 compliant system a ‘file’ was limited to two levels of hierarchy beneath it (file/sub-file/parts). In a Moreq 2010 compliant system an aggregation can have any number of levels of hierarchy. The MoReq 2010 ‘aggregation’ has a different relationship to a business classification/fileplan than a MoReq 2 ‘file’. The MoReq 2 file sat at the very bottom of the business classification/fileplan. An aggregation can be a multilevel hierarchy in its own right, so it can sit separate from the business classification, whose role is not so much to act as the only means of navigating around the records system, but more to apply retention rules to records.

• the idea of one monolithic corporate business classification/fileplan universally applying retention and access rules to all records In previous versions of MoReq only one classification could be linked to retention rules and used to apply retention rules to records (a hierarchical corporate business classification, also called a fileplan). In Moreq 2010 compliant systems there is the possibility of having several classifications, any number of which can be used to apply retention rules to records. If a record is classified against more than one classification then one of them should be nominated as the ‘primary classification’ for that record. The primary classification is the one that the record inherits its retention rule from.

• the idea of their being a single type of ERM system that can meet the needs of all organisations in all sectors Jon Garde identified that MoReq 2 was twice as long as the first MoReq. Jon says the reason for this is that both MoReq and MoReq 2 tried to specify systems that could meet the records management needs of all organisations, whatever their sector. This meant that if any sector of the economy could prove they had a genuine records keeping requirement then MoReq would try to incorporate it into the specification. The disadvantage of this is that vendors had to configure their systems to cover all sectors and all eventualities, which pushed up the cost for them of developing compliant systems. MoReq 2010 has been written so that the core module contains only requirements that are common to all or most organisations. If a sector has specific requirements they are able to write a separate MoReq 2010 module to capture those needs. Vendors that wished to target that sector could add that functionality to their system and ask for it to be tested and certified against that module. Vendors that didn’t wish to target that sector could ignore it. Jon Garde argues that the MoReq 2010 will be more sustainable over time than previous specifications, as new needs can be incorporated into new modules without having to republish the whole specification. Marc Fresko gave the counter argument that the structure of MoReq 2010 will be more complex for records managers in organisations to work with, because they are going to have to decide which modules are important enough for their organisation to insist upon.

The new aspirations of MoReq 2010

In place of those abandoned aspirations MoReq 2010 brings in three new aspirations:

• the idea that the electronic records management system will sit behind the existing applications used to create and capture records in the organisation A system can be MoReq 2010 compliant without having any user interface. The core module merely insists that if a system does not have a user interface it must have an API (application programming interface) to enable it to integrate with those systems in the organisation that users do interact with to create records. Note that integrating an electronic records management system behind existing content repositories poses very different challenges to those posed by implementing a stand alone EDRMS that end-users directly engage with to save their records. It is likely that your various content repositories will have been procured from various vendors, written in various programming languages, and have varying quality APIs. Each integration to each content repository is a project in its own right. Aside from the technical challenge there is also the semantic challenge – each of those content repositories will have their own way of organising and describing content.
• the idea of MoReq compliant systems controlling records held in other systems a system can be MoReq 2010 compliant without having the ability to store records, provided it has the ability to apply classifications, access rules, and retention rules to records stored in other systems; provided it can protect those records from deletion or amendment; and provided it can maintain (and export) an audit trail of events that happen to those records.
• the idea that when an organisation changes its records system the new records system could understand everything that had happened to those records in the outgoing records system. Record systems capture records and apply retention rules to them. Jon Garde points out that the span of years of many of these retention periods is longer than the life of most records systems. Jon wants a MoReq 2010 compliant system to be able to export a record and its associated metadata at any point during the retention period, so that the receiving system could understand what retention policy has been applied to the record, when that retention period started, and how long it had left to run. The new records system could then apply the remainder of the retention period. MoReq 2010 is much stricter than previous specifications on how a system maintains its audit logs. The system must be able to export an event history for every object in the system.  The event history must record everything that has happened to that object since it was created in or captured by the system. In order for this event history to be understandable by another system MoReq 2010 insists upon the use of unique identifiers. A MoReq compliant system needs to allocate unique identification numbers to every type of object (records, aggregations, retention policies, classification headings etc.); to every actual object; to every type of event in the system (for example ‘the title of a document is changed), and to every actual event.
• the idea of line of business system vendors incorporating records management functionality into their systems By stripping down the core requirements to a very minimum the DLM-forum hope that providers of line of business systems will see that it is possible for them to seek MoReq 2010 certification.  The hope/dream is that providers of finance systems, human resource management systems, facilities management systems, and customer relationship management systems will add MoReq 2010 functionality to their applications . Note that vendors of those kinds of applications won’t start to think about MoReq 2010 compliance unless and until purchasing organisations start specifying a preference for MoReq 2010 compatible applications. The challenge here is that records managers often do not have a voice in those kind of procurement exercises.

The likely impact of MoReq 2010

MoReq 2010 will bring interesting effects for records managers. The standards we were used to in the first decade of the 21st century (TNA 2002, DoD 50.15, MoReq 2) all ended up certifying a very similar type of system, implemented in a very similar way. Not quite ‘seen one EDRMS seen them all’ but almost.

Out of MoReq 2010 will sprout a variety of different applications, which will need a variety of different implementation methodologies. This is a challenge for the wider records community, not the sponsors of MoReq 2010. All MoReq 2010 can do is to encourage vendors to provide applications that manage records created and/or held in different content repositories (and provide certification to verify that they have that capability).

Whether or not MoReq 2010 ends up being successful will ultimately depend on whether we records managers can come up with viable implementation approaches for the type of application that it will certify.

I spoke about SharePoint to an audience of intranet managers yesterday. The talk was at J Boye’s ‘SharePoint at work’ event in Copenhagen.

I made the following points:
• SharePoint is not the first platform that an organisation could use both for their intranet and for their general collaboration/document management environment. But it is the first platform to be market leader in both spheres.

• Managing a collaboration/document management environment is a different ball game from managing an intranet environment.

• The scale is different. In an intranet environment you can expect a sub-section of the organisation to contribute a small amount of content on a monthly basis. In a collaboration/document management environment you may reach the point where most of the organisation are contributing on a daily basis.

• The risks are different. In the collaboration/document management environment there will be material with sensitivites (personal/commercial/political). Such material is not typically contributed to intranets.

• The difference between the functionality of SharePoint when used for an intranet and the functionality of SharePoint when used for collaboration is paper thin. In either case you store content within SharePoint site collections. If you intend a site collection to be used for the intranet then you use the publishing site template to create the top level site of the site collection. If you intend a site collection to be used for collaboration you create the top level site with a different template (for example a team site template).

• Organisations typically put site collections intended for the intranet in a separate web application to those used for collaboration, with a separate URL address, different branding and different administrators and administration settings.

• There is no licensing difference between SharePoint Server 2010 used for an intranet and SharePoint Server 2010 used for collaboration/document management. Where an IT department have purchased SharePoint Server 2010 licences (for example as part of an enterprise agreement with Microsoft) it is hard for either intranet managers or records managers to explain why they want to buy and use a different system

• There may well be synergies to be derived from having both intranet and document management happen in the same environment, but realising these synergies is not easy. An organisation might identify valuable type of information (for example research reports), define a content type for it and then surface such material from the collaboration environment onto the intranet.

• If an organisation wants to use content types to leverage and make the most of valuable types of information then it becomes very important to control their content types. There is a tendency for content types, like everything else in SharePoint, to sprawl. Content types can be created in any site by any site owner, and be available not just in that site but in all sites underneath it. The more content types you have the less chance there is of a team choosing appropriate content type(s) for their document libraries.

• Your records manager may want to use content types to apply retention rules and/or to route documents to a records centre. If so they will want to apply content types to large aggregations of documents (record series such as ‘refurbishment project records’ ‘client case file records’ ‘consultation records’ etc.) rather than to individual document types (‘project initiation documents’ ‘research reports’ ‘meeting minutes’ etc.)

• There is no way of using content types for both record series aggregations and for document types (because a document can only belong to one content type at a time).

Question from the audience about document sets
I was asked whether ‘document sets’ (a new kind of content type available within SharePoint 2010) could be used to aggregate different content types. I said that although document sets do aggregate content types, I did not think they would not work for records management purposes.

Document sets were intended by Microsoft for use with compound documents (such as ‘health and safety manuals’).

In theory document set content types could allow a records manager to:
• define a document set content type for a record series (for example ‘refurbishment projects’)
• enable content types to be used within instances of that document set (for example ‘architectural drawings’ ‘contracts’ ‘contractual correspondence’ ‘meeting minutes’).

The problem is that if a colleague uploads a document to a ‘refurbishment project’ document set and gives it the content type ‘meeting minutes’ then it would take its retention rule from the information management policy defined for the ‘meeting minute’ content type and not from the policy defined for the ‘refurbishment project’ document set content type.

Facilitated discussion on SharePoint governance
Earlier in the day I had the pleasure of facilitating a round table discussion about SharePoint governance. Two intranet managers told me that their organisations had decided to apply strong governance arrangements in the site collections used for their SharePoint intranet, but to be much less controlled in the collaboration environment, where team sites would be given to local teams on demand.

We talked in the group about the ‘Pontius Pilate’ attitude to SharePoint of some IT departments who treated SharePoint as though it was a tool set. They provide team sites on demand, and when it grows out of control they say it is the fault of the business for not governing it properly.

Someone else in the group said that the trouble with governance is that at first it doesn’t seem like a big issue, you have a system that you want people to use, there are not many users yet, and there is not much content yet. Later on, when the system has grown and evolved you realise the need for governance, but trying to put governance in place is much more difficult (like locking the stable door after the horse has bolted).

I said that governance in SharePoint is primarily about taking away powers (for example the power of site owners to create sub sites, and the power of site owners to create content types). There is a resource implication to taking away those powers. You need to give people a quick way of obtaining new sites and new content types where appropriate. And that means having people in place who have the ability to judge whether or not the new site or content type is necessary and appropriate. If requests for new sites/content types are granted on demand then you have not added any value.

On 16 February this year I heard Mark Field describe his implementation of SharePoint for collaboration and records management within the UK Department for Education (DfE). Mark’s talk was at a records management update seminar run by Unicom.

Mark’s implementation has the potential to be extremely influential.   It currently has a userbase of 3,000 people within the Department.  DfE are planning to offer their SharePoint facility as a service via the cloud – firstly to the UK education sector, and later to the wider public sector via the Government Application Store on the G-Cloud.  There is currently no timescale as to when it will be available on G-Cloud (when pressed Mark said ‘hopefully within our lifetimes’).

DfE are using SharePoint team sites (rebranded as ‘Information workplaces’). They use the SharePoint records centre for records management, and have implemented SharePoint for their intranet.   They are currently using SharePoint 2007 but will move to SharePoint 2010 once they have transitioned their service to the cloud.

Governance controls

The Department has tight governance controls in place to control the spread of  SharePoint team sites.  Only their Information and records team can create team sites (information workplaces).   Each team site/workplace must have a ‘Workplace owner’ who signs an agreement acknowledging their responsibilities in managing the site.  They have 200 Information workplaces, which tend to be based around a particular work matter or policy issue.

Records management

Regular ‘sweeps’ are undertaken of team sites to suck documents needed as records, and route them to the records centre.  The sweep captures documents that meet all of the following three criteria:

• at least a year old
• a major version (1.0, 2.0 etc)
• checked in

When a document is routed to the records centre it leaves behind a link in the team site document library from which it came.

The Department has decided not to implement a functionally based fileplan. The record centre is structured to mirror the organisational structure.  Retention rules are linked to content types. The Department has limited itself to 11 content types.

There was a mixed reaction by the audience to the model adopted by DfE.  Some people (including ex `Information and Records Management Society Treasurer Alison North) praised its simplicity.   Others were concerned by the Department’s rejection of a functionally based fileplan in favour of the organisational structure.  The question was raised as to whether organisational churn would render the records centre difficult for future users to understand and navigate.

Mark defended the decision to use the organisational structure to organise the SharePoint records centre.  He said that all a searcher would need to know to find documentation is what the relevant organisational unit was called at the time of the document’s creation.

Replacement of the DfE’s Electronic Document and Records Management System (Meridio)

Mark’s lack of faith in a functional fileplan is based on his experience at a previous organisation, where users ‘ran a mile’ from a multi-level fileplan he had constructed with colleagues; and from DfE’s own experience with using a functional fileplan on a  traditional (TNA 2002 compliant) EDRMS system (Meridio).

Mark said Meridio had good functionality but it  was only adopted by 20 to 30% of the Department.   The  Department have ended their EDRMS implementation, and imported all records from Meridio into a SharePoint records centre (a separate records centre from the one that they are using to handle content generated in SharePoint).

DfE’s application of retention rules

The department are choosing to link retention rules to SharePoint content types, and to limit the number of content types to 11.  This is a radical example of the big bucket approach to implementing retention rules.   In most other implementations of ‘big bucket’ retention rules the organisations concerned have applied between 50 and 100 retention rules to their organisation’s information.  Applying only 11 retention categories seems  low to me.

One of the content types Mark mentioned was for human resource documents.  I presume that a ‘human resource’ content type would need to embrace (among other things) both:

• all the documentation arising from recruitment campaigns
• documentation of dealings with individual employees

Most organisations keep records of their dealings with individual employees for a considerably longer period than records of their recruitment campaign.

My assumptions as to the nature of the Department for Education’s SharePoint records centre

It is difficult for me to accurately picture Mark’s model without having seen screen shots.  My assumptions are that:

• the records centre contains one document library for every organisational unit that has existed since the start of their SharePoint implementation. Each library contains all the documents routed to the records centre that originated from colleagues attached to that unit
• when organisational churn occurs a new records centre document library would be created to accommodate records routed from that organisational unit (although it might be a year before any documents arrive to fill it!)
• when documents move from a team site to the records centre they bring with them important contextual information such as the details of the pathway/breadcrumb trail in which they had been stored ( team site/document library/folder/sub folder)
• a visitor to the records centre would be able to sort each document library by any column of metadata, including which of the 11 content types a document belong to, and which team site it came from.

Does the records centre add value?

The big challenge for any implementation of the SharePoint  records centre is the question what value does the record centre add?   What value does it add to have documents moved from one logical structure (Team sites, arranged by policy issue or matter) to a different logical structure (A records centre arranged by organisational unit)?

Is there any reason why anyone would prefer to look at the records centre rather than the original team site? (I can’t think of one).  All the time that he original team sites are maintained that is not a problem.  But how long will the Department retain its  team sites?

I suspect that the Department for Education’s intention would be to apply a shorter retention period to the SharePoint team sites than to the contents in the records centre.  Once the original team sites are deleted you are left only with the content of the records centre, kept in a different arrangement to that known by the people who worked with the documents at the time that the work was carried out.

Take the example of a document library within a team site set up for three different organisational units to collaborate around policy on special need provision in schools.   How easy would it be for a searcher to virtually reconstitute the contents of that team site document library once the contents have been sent to different records centre libraries and after the original team site has been deleted? Does it matter that other information held in the team site, but not captured in a document library, will not have been sent to the records centre?

Practical reasons for moving records from team sites to the SharePoint records centre

Later in the same Unicom event Sharon Richardson talked about the difference in storage capacity of a site collection containing collaborative team sites, and a records centre:

• Microsoft recommends that site collections containing collaborative team sites are not allowed to grow about 100GB (which Sharon estimates is only 200,000 documents). The constraint is due to the impact of a large site collection on other site collections held on the same SQL Server content database
• A records centre can be allowed to grow much larger provided that its’ content is stored in a dedicated SQL Server content database which contains no other site collections

The records centre can mitigate problems caused by the quirks of SharePoint’s site collection architecture – but I am still not sure that this counts as adding value!

I intend to set up a SharePoint Server 2010 demonstration environment, to enhance my training and consultancy.  There are many different options for how I go about this.   I could host it myself.   I could get it hosted by Microsoft, or a third party.  I could install it on an Amazon Web Services EC2.  Below are the pros and cons of these different options, with reference to my situation at this point in time.

Option 1: Install SharePoint Server 2010 on my mac
The first option I looked at was to build a SharePoint Server 2010 environment on my laptop. The fact that my laptop is a mac is not necessarily a show stopper.   There are plenty of discussion board postings from people who have set up development or test environments of SharePoint Server 2010 on MacBook Pros (for example this thread)

SharePoint needs the whole Microsoft stack to run on (Windows operating system, MS SQL Server database, Internet Information Services, and the .Net programming framework). None of these things run on the mac operating system. However that in itself is not too much of a problem. I could purchase virtualisation software (for example VMWare Fusion) to run on the mac. I could install Windows Server 2008 R2, IIS, SQL Server and .Net onto the virual machine, and could then install SharePoint Server 2010.

I would have to shell out for the licences to those products. The best way for me to purchase the licences is via a Technet Professional subscription which currently costs £234 per year.  A licence for VMFusion is currently priced at $79.99.

So far so good, but then I hit a sticking point.  Shane Young in his article ‘Building a Perfect Test Environment for SharePoint 2010’  in the book Real World SharePoint 2010 recommends giving a virtualised test/demo/dev environment of SharePoint 2010 at least 4GB of RAM.  I use a Macbook Air which only has 2GB of RAM. The Air is designed to be lighweight so certain compromises have been made with the design.   One such compromise was that the Air does not allow expansion or upgrade of the RAM.

The option of hosting SharePoint Server 2010 on a device of my own is still open to me if I purchase a new laptop. Aside from the additional cost, this option has the disadvantage of giving me an extra machine to carry around with me. I wanted to avoid that if possible so I decided to look at the cloud options.

Option 2: Software as a service (Microsoft’s SharePoint online offering).
Of all the options open to me this would be the most straightforward. No licences to buy or software to install. Microsoft offer SharePoint online in combination with their online versions of Office and Exchange.  This combined service is currently running under the name of BPOS.   Unfortunately the SharePoint component of BPOS is SharePoint 2007, and I want SharePoint 2010.   BPOS doesn’t have SharePoint 2010 because Microsoft are replacing it with a new service, called Office 365, which does include SharePoint 2010.   Office 365 is in beta at the moment.  This should be an advantage – (because it is free of charge whilst it stays in beta). However there is a waiting list to get onto the beta, and I have no way of telling when or whether I will be provisioned with a beta site.

Option 3: Third party hosting
There are plenty of vendors that offer hosted SharePoint via the cloud. I contacted one of them, Rackspace, and asked if they would offer a SharePoint Server 2010 installation for me. It would have been OK if I had been happy with SharePoint Foundation. SharePoint Foundation is a free downloadable version of SharePoint. It has the basic collaborative features but lacks important features such as My Sites. For the demonstration environment to be any use to me I need SharePoint Server 2010. Rackspace told me that if they were to provide me with SharePoint Server 2010 they would need to kit me out with a dedicated server, and that would cost me $736 per month (which would be like taking out another mortgage!).  I told them my storage requirement would be very very light, and asked why they couldn’t host my SharePoint environment on a virtualised machine on an existing server. They said they couldn’t do that because of Microsoft’s licensing requirements.  It seems like the other vendors are under the same restrictions – look at these monthly prices from FP web

Option 4: Infrastructure as a service: installing SharePoint on Amazon EC2
The final option I have looked at is installing SharePoint Server 2010 on an Amazon EC2 instance. Tristan Watkins has a very useful blogpost about the pros and cons of hosting SharePoint on EC2. In it he explains how to go about setting up SharePoint 2010. The process involves:

• selecting an Amazon Machine Image which has Windows Server 2008 already installed on it (with the Windows license fees included in the fee),
• installing MS SQL Server, the rest of the Microsoft stack, and SharePoint 2010 (you would need your own licences for those products)

Here is the full process as described by Tristan:

• Launch an instance from the default Windows Server 2008 image.
• Install SQL and SharePoint (this should be possible at just under 30GiB).
• Configure stuff and shut down the instance.
• Take a snapshot.
• Create a new volume at 50GiB based on the snapshot.
• Detach the existing volume from the instance and attach the new volume.
• Create an image from the instance.
• Launch the existing instance and create additional instances from the new AMI as needed.

This looks to be a very complex process, and the pricing model looks complex too.  But it seems far more affordable than the 3rd party hosted services, even allowing for the fact that I would have to buy licences for the products in the Microsoft stack.

Conclusion
If it wasn’t for the fact that I am a MacBook Air user I would have installed SharePoint Server 2010 on my laptop and not looked at the cloud options.  Cloud computing is great when you either

• want to avoid the trouble of installing and configuring software yourself
• have a storage requirement that is hard to predict
• want to avoid buying new hardware

Only the third of these applies to my situation.  I am happy to install the software myself because it would be a useful learning curve for me.  My storage requirement is predictable –  I will put very little stuff into the SharePoint demo environment   But I do want to avoid buying (and having to carry around) a new laptop.  And for that reason I intend to try the Amazon route.

I enjoyed speaking about SharePoint at the Online Information Conference 2010 at Olympia this Wednesday.

Martin White spoke after me.  He said that ‘SharePoint met 80% of most people’s ECM needs.  The problem comes if something you need fits into the 20% that is missing’.

A member of the audience pointed out that there are lots of plug-ins available to fill the gaps, and many of them are relatively cheap in terms of license costs.  True. But how many plug-ins can your organisation afford to purchase, integrate and support?  What is the impact on the time it takes to implement service packs and upgrades?

From a records management point of view there are some fairly fundamental things in that missing 20%.  I spoke to an organisation recently who declined to use SharePoint as their electronic records management system, despite already possessing the licences for it, because the necessity for customisations and plug-ins outweighed the fact that they did not have to pay for licences.

They  identified that they would need plug-ins for :

• e-mail integration (the facility that enables someone to select an e-mail in their Outlook client and choose a specific location within SharePoint to save it to).
• export (the facility to export documents and other content, with all their associated metadata, in order to migrate the content to another system)

These two features are not marginal, they are fundamental features for an electronic records management system.  At a recent DLM forum meeting I heard Jon Garde state that export was one of the most important features of the draft MoReq 2010 electronic records management system specification, because the span of time that records are needed for is usually longer than the lifespan of the application in which they are stored.

This afternoon I published the following article to the AIIM ERM community:

In June I saw Peter King, Office Server Group Manager of Microsoft UK, give a keynote speech to the AIIM roadshow in London. One of Peter’s slides contrasted functions that SharePoint 2010 could fulfill on its own, with functions that SharePoint needs the help of third party applications to fulfil. Records management was listed as being one of the things that SharePoint 2010 could do on its own.

However Microsoft say that they have no intention of submitting SharePoint 2010 for certification against one of the recognised standards for records management systems, for example DoD 5015.2 in the US or MoReq 2 in the European Union. They advise customers wanting that level of records management functionality to look to third party products.

SharePoint 2010 can do a form of records management, but it is records management as defined by Microsoft, not as defined by the international records management community. The fact that Microsoft are using their own definition of records management is not in itself a bad thing – we need innovation in this space, and vendors slavishly following worthy specifications drawn up by committee isn’t going to do anyone any favours. But it does mean that we need to closely scrutinise, and debate, the records management functionality contained in SharePoint 2010.

The lack of a clear records management model in SharePoint 2010

With SharePoint 2007 Microsoft had attempted to make two innovations in records management:

• They wanted people to use ‘content types’, rather than folders as the primary means of organising and managing documents and other content
• They wanted documents to be moved into a records centre to be treated as records, rather than treated as records inside the team sites that people worked in

In SharePoint 2007 there was a clear records management model – the only problem was it didn’t work. Microsoft had given all the power to the content type and none to the folder, but the content type on its own proved unequal and unsuited to the task of managing records.

In SharePoint 2010 Microsoft are hedging their bets:

• Content types have been adapted to make them more suited to managing records (through the addition of a new kind of content type called a ‘document set’). But at the same time folders have been given more functionality
• Record centres have been improved, but Microsoft are also providing an ‘in place’ records management facility to enable people to protect records from deletion or amendment without moving them to the records centre

In SharePoint 2010 there is no single records management model. Instead there are different choices and options. SharePoint 2010 shows some improvements in records management functionality when compared with SharePoint 2007. But the lack of a clear records management model in SharePoint 2010 means it will not be easy for implementers to decide upon and to sustain a coherent approach to records management.

For example lets imagine you intend to deploy a fileplan in SharePoint. I am defining a fileplan as a hierachical classification of the work an organisation carries out. There are three different ways you could deploy that fileplan. You could deploy it as:

• the site collection/ site/ sub-site hierarchy
• a hierarchical structure within a records centre (or spread across several records centres)
• a taxonomy in the managed metadata service

Each of these options has its pros and cons.

Using the site collection/site/ sub site hierarchy to house your fileplan (‘In place’ records management)

Whereas in SharePoint 2007 you had to send a document to the records centre if you wanted it protected from unauthorised deletion or amendment, in SharePoint 2010 you can have it protected within the team site.

There is a ‘Declare Record’ button on the SharePoint document library ribbon that enables colleagues to save a document as a record. Once a document is declared as a record it is protected from amendment or deletion.

In theory this option means we can dispense with the records centre(s).

However having content protected as a record is only one aspect of how records managers want content needed as a record to be treated. We also want to to ensure that content is organised within a managed structure. A structure that persists, and continues to make sense, over time.

The problem with managing records ‘in-place’ is that it is very difficult to control the structure within which content resides.

Take a typical piece of content in SharePoint – a document, sitting in a folder within a document library. The library sits within a sub-site, which sits within a site, which sits within a site collection.

The structure that a document resides in is the breadcrumb trail encapsulated in the URL of the site, the library and the document itself . The breadcrumb trail starts with the name of the site collection and carries on with the name of the site, sub-site, document library and folder.

In theory you could impose the rigidity of a hierarchical records fileplan onto this structure. It is possible to:

• set up a SharePoint site collection for each of the top level headings in your fileplan
• set up a site within those site collections to represent the 2nd level of your fileplan, and sub-sites beneath that for the third level of the fileplan. If you needed another level of hierarchy you would add another level of sub-sites
• ask each team to create a new document library for each new piece of work that they start, within a lowest level sub-site

However in practice such rigidity may become too much of straitjacket for a SharePoint implementation. If SharePoint becomes popular in your organisation then sub-sites and document libraries will be created for all sorts of reasons, some with the intention of holding records, but others for ephemeral collaboration. It isn’t necessarily going to be meaningful to shoe-horn them all underneath a fileplan hierarchy.

On the other hand if you do not think through the hierarchy of site collections/sites/sub-sites, and allow a free for all in site and sub-site creation, then you end up with a haphazardly evolving structure, a sprawl. When that happens the in-place records management feature is of little use, because your declared records are randomly dotted around the massive SharePoint sprawl like tiny islands in the Pacific Ocean.

Housing your fileplan in a SharePoint records centre

The way the records centre works in SharePoint 2010 is broadly similar to the way that it worked in SharePoint 2007 (although to say that the records centre worked at all in SharePoint 2007 is in my view something of an exaggeration).

• In SharePoint 2007 the idea was that knowledge workers would click on a document in a SharePoint document library, and would select the option ‘send to the records centre’. In the records centre the ‘records routing table’ would route the document to the appropriate records library, providing the document belonged to a SharePoint content type, and provided that a records routing rule had been set for that content type.
• In SharePoint 2010 the records routing table is replaced by the content organiser and the drop off library. When someone selects the option ‘send to records centre’ documents are received by the drop off library, and then processed by the content organiser. The content organiser sends the document to the appropriate records centre library, or the appropriate folder within a records centre library, providing that the document belongs to a content type or a document set, and providing that a routing rule has been set for the content type or document set that the document belongs to

In terms of configuration in SharePoint 2010 you can have a drop off library and content organiser within the records centre itself (like the records routing table in SharePoint 2007). You can also have content organisers and drop off libraries inside each collaborative site, that route documents to the records centre or to particular places within the records centre.

The improvements made to the records centre between SharePoint 2007 and SharePoint 2010 are incremental rather than revolutionary, but they are worth listing:

• Routing to a specific folder – In SharePoint 2007 the fact that you could have a folder structure within a records centre library was rendered useless by the fact that the records routing table could only route a document to a records library, it could not route a document to a specific folder within a records library. In SharePoint 2010 the content organiser can route a document to a specific folder within a records library in the records centre
• Inheritance from folders– Folders have more functionality in SharePoint 2010 (they had none in 2007). You can set retention rules on high level folders, and you can allow all the sub-folders and documents saved within it to inherit the retention rule. This means you can get a folder structure in a records centre library to function like a fileplan
• Routing on the basis of metadata properties In SharePoint 2007 the routing table could only apply a rule to a content type. In 2010 rules can be set rules at a more granular level. You can base a rule on a value in a metadata column field of a content type. For example you could define a column entitled ‘project’ for a content type/document set. You could set up a controlled vocabulary for the ‘project’ metadata column. You could get the content organiser to look at the value in that column for any incoming document, and route it to the folder in the records centre for that particular project.
• Multiple records centres – In SharePoint 2007 organisations were only able to have one records centre. In SharePoint 2010 you can have as many records centres as you wish. Rather than having one records centre with one huge fileplan, an organisation could choose to spread its fileplan over several records centres in SharePoint 2010

The whole concept of the record centre in SharePoint brings with it three big disadvantages

• It requires you to use content types – If you are using the records centre, you are going to have to use content types or document sets. This is because the content organiser works by applying rules to content types/document sets, not to folders. Content types and document sets are more powerful than folders, but they are also more complex for you as an administrator to set up and maintain, and for your colleagues to understand and use.
• You have the administrative overhead of maintaining all the routing rules – Most organizations are used to having some rules to automatically route some documents (particularly where they have workflows set up to automate some important business processes, or within some line of business systems). However I don’t know any organisation that is used to setting rules for the routing of documents across all its business processes. This reliance on pre-defined routing rules for the basic records management process is new to SharePoint – none of the established electronic records management system worked that way. I am yet to be convinced that it is feasible to set up and maintain routing rules covering a large organisation’s entire range of activities
• there is no guarantee that the records in the records centre will be useful– Microsoft envisages collaboration sites as being the sphere of the knowledge worker, and records centre sites as being the sphere of the records manager. But if knowledge workers have little or no interaction with the records centre, then in what sense can we regard the accumulation of documents that build up there as an authentic record of the work that those knowledge workers carry out?

Installing your fileplan as a taxonomy in the Managed Metadata service

The third option for your fileplan in SharePoint 2010 is to have it as a controlled vocabulary, a three level taxonomy that sits in the Managed Metadata Service, and that can be used to:

• populate metadata columns in document libraries

and/or

• populate columns of metadata for content types and/ document sets

Support for hierarchical taxonomies is a new feature in SharePoint 2010. SharePoint 2007 only supported flat controlled vocabularies (akin to a drop-down list of headings).

A taxonomy in SharePoint’s managed metadata service can only go three levels deep (the three levels are called: groups, term stores and terms). This is a significant constraint – in my experience those organisations of around 800 people that have corporate fileplans tend to have a four level classification (which they use to organise folders that are created when a piece of work starts).

The weakness of having your filepelan as a taxonomy in the managed metatdata service is that you cannot link retention rules to SharePoint taxonomy terms. You can only link retention rules to content types, document sets, libraries and folders. You would still need to decide with this option whether to move records to the records centre, or whether to manage records in-place in the collaborative team sites.

If you are using the in-place records management option, then the advantage of having the fileplan as a taxonomy is that it removes the need to warp the site collection/site/sub-site structure into the form of a fileplan.

It is not easy to enforce a fileplan set up as a SharePoint taxonomy, particularly with the in-place records management option. This is because SharePoint gives teams a lot of options on what type of object to use when collaborating on work. It is hard to predict whether any particular team will want a sub-site, a document library, or a folder for their piece of work. This makes it harder to issue a blanket policy rule such as ‘every document library must be classified against the fileplan in the managed metadata service’

One option might be to use content types, and specifically to use the SharePoint document set content type. In theory it is possible for an organisation to use ‘document set content types’ as a fourth level fileplan classification, with each ‘document set content type’ linked to a specific pathway in the three level fileplan in the managed metadata service. The downside of this approach is that it would force people to use document sets to manage documents of their work. Document sets are a new feature in SharePoint 2010, and at the time of writing I have not seen any case studies from organizations using them in practice.

Conclusion

SharePoint is now a mature collaboration system, having gone through several iterations of the product. But the records management in SharePoint is not yet mature. The records centre in SharePoint 2007 could not be made to work to any scale without extensive custom coding. Many of the concepts on which records management is based in SharePoint (content types, document sets, routing rules, and the records centre) are unique to SharePoint. Records management in SharePoint is still unproven, and will remain unproven until we know of a reasonably sized organisation who have successfully used it for records management across their full range of activities without extensive customisation.

For these reasons I think it is important that we as a records management community look on SharePoint 2010 with a sceptical, questioning eye.

Yesterday I gave a joint presentation in Sheffield on SharePoint in higher education to IWMW10 (an annual workshop for people running websites in UK higher education institutions). I was presenting with Pete Gilbert of the University of the West of England. There is no love for SharePoint amongst web professionals in higher education, which made the reaction to the talk very interesting (and in particular the Twitter backchannel under the hashtag IWMW10) .

Before I talk about the backchannel’s reaction, I’ll describe some conversations I had in the tea break immediately after the talk.

Pressure on web teams to use SharePoint as a content management system
Two people from two different universities came up separately to tell me the same story: they are running a web site, and they have experienced unwelcome pressure from their IT departments to use SharePoint as their web content management system (CMS):

• One of the webmasters is working at an institution which has abandoned a selection process to choose a web content management system. Instead their IT department, in conjunction with the Marketing department, has decided to pick SharePoint as their web CMS, without involving the wide group of internal stakeholders that had been part of the abandoned selection process.
• The other webmaster responded to similar pressure at their institution by dedicating time to finding out about SharePoint and successfully making the case that SharePoint was not the right Web CMS for them

Oxford University’s SharePoint implementation
Ian Senior from Oxford University updated me on their implementation of SharePoint as an internal collaboration system. SharePoint provides a great many configuration choices for owners of team sites. Oxford have sought to reduce this complexity by narrowing the scope of their implementation. They are concentrating on rolling out SharePoint to research teams, and to committees, and have devised template team sites tailored for those particular pieces of work.

Edinburgh Napier’s SharePoint implementation
David Telford from Edinburgh Napier University came up to discuss their SharePoint implementation. Napier are the nearest thing to a SharePoint flagship in UK higher education. They are running SharePoint for their external web site, their internal web site, as a student portal, and to provide collaboration sites for staff. This is supported by a team of four .Net developers. David says that on average two Universities a week come up to Edinburgh to see what they are doing with SharePoint. This creates a pressure on web teams in other institutions, as some of the visitors to Edinburgh Napier return to their own institutions expecting their web teams to replicate the Napier example, using SharePoint.

My position on SharePoint
The Twitter back channel during our talk was unanimously hostile to SharePoint, and gave some very valid criticisms of the product.

I am used to hostility to SharePoint, because my home milieu is the records management community, where most people dislike SharePoint profoundly. However people in the records management community are usually hungry to hear about SharePoint, whether that is because they wish to marshall arguments against it, or to find ways of mitigating the weak records management functionality in it. I got the impression at IWMW10 that there was hostility to there even being a session on SharePoint at the conference, which surprised me.

In the face of such a reaction it is hard not feel a pressure to take a position for or against the product. I have no interest in taking a position for or against SharePoint. My profession is as a trainer and consultant and my job is to help people understand complex phenomena so that they can make decisions about them. I talk about SharePoint because it is there. There are some moral issues in life you have to take a stand on, but I don’t see SharePoint as one of them. I have seen plenty of reasons to be cynical about the SharePoint phenomena, but also reasons to admire some of the things that fellow professionals have done with SharePoint.

Reasons to be cynical about SharePoint
I have seen an IT director bulldozing through SharePoint too quickly and without thought or preparation in order that he could add ‘rolled out SharePoint to an entire organisation’ on his CV. I have seen IT departments undermine established document management systems in organisations by simply signing enterprise agreements with Microsoft and rolling out or making available SharePoint team sites.

I agree with the person I spoke to last year who questioned the ethics of Microsoft promoting SharePoint 2007, which had no support for accessibility standards, for use as a web content management system in higher education.

In my own professional area, records management, Microsoft themselves have behaved cynically. They released SharePoint 2007 with records management functionality that was half-baked and not ready for use. Instead of admitting its inadequacies, they continued to claim that their product could be used for records management purposes. Microsoft contracted an external organisation to write some code that they did not support, and that would prove useful to no-one, purely in order to gain certification for SharePoint 2007 against the main US records management standard.

Prior to the release of SharePoint 2007 most IT departments were not interested in what content management their organisation used, or which collaboration or document management system was chosen. The relevant professionals in those areas (web teams, knowledge managers, records managers) would normally be able to take the lead on that. As soon as Microsoft, with its established relationship with IT departments, issued an enterprise content management system of their own in the form of SharePoint, you began to get IT departments promoting SharePoint. The fact that SharePoint client licences are available to IT departments bundled up with Window, Exchange and Office effectively pulls the rug from under the feet of professionals wanting to chose an alternative system.

There is a legitimate debate about whether or not SharePoint can be regarded as free. You still have to pay for servers, for server licences, for SQL server licences, and if you want to use it for an externally facing web site or extranet you have to pay for an external connectivity licence. And of course you are paying for the bundle itself in the form of the Microsoft Campus Agreement or Enterprise Agreement. But the fact remains that is perceived as free or very cheap by organisations who would be signing that agreement with Microsoft anyway.

Reasons to admire things that people are doing with SharePoint
Despite my cynicism about some of the actions of some IT departments in relation to SharePoint, I also find some uses of SharePoint interesting and even inspiring. I admire implementations like those at Imperial College and of Pete Gilbert at UWE. They both started small with SharePoint, built up knowledge of the product, and offered it as a solution where teams have come to them with a particular need or problem. These implementations have tended to concentrate on SharePoint for internal collaboration, rather than SharePoint as a web site. I also admire David and his team at Edinburgh Napier and their pride in the fact that they have made a tool, that many find difficult to use, fit the needs of their institution.

What I am admiring here is not SharePoint as a tool. It is the professionalism of people like Pete at UWE, David and his colleagues at Edinburgh Napier, and of the team at Imperial. This professionalism manifests itself in their efforts to get to know SharePoint, and in their disinterested and honest advice to colleagues on the circumstances in which SharePoint would be (and would not be) useful to them. That professionalism would be equally admirable if Napier were using Plone as a web content management system, or if Pete at UWE was rolling out Lotus Notes.

That professionalism and care is a million miles away from an IT department choosing SharePoint for the sake of it, which is a practice I deplore.

We are only a decade into the 21st century, but it is already apparent that there will be something different about the records produced by this century, as compared to those of any other time period.  It is not just that they are going to be in electronic rather than hard-copy form, it is more fundamental than that.

Records in previous centuries

The United Kingdom’s National Archives in Kew houses an unbroken eight hundred years worth of records. In every previous one of those 8 centuries of English/British record keeping, there has been a common method of keeping records.  A ubiquitous method that was used for record keeping by any and every organisation.

• In the 13 and 14th centuries the ubiquitous method of record keeping was the roll.   Manorial courts, ecclesiastical courts and the royal administration all kept records in the same way. They would issue a communication (a charter or a judgement for example). Their clerks would copy the communication out onto a strip of parchment and sew it onto the end of the copy of the previous communication, creating a roll
• By the nineteenth century the ubiquitous method of record keeping was the register and the book. Government ministries, branches of the armed forces, companies and trading houses all kept records the same way.  Clerks would keep a register of all incoming and outgoing letters, with one line of description for each letter. Outgoing letters would be copied into a letter book before they were sent out.  Incoming letters would be bound into books, or gathered together in boxes or files.
• In the twentieth century the ubiquitous method of record keeping was the file. Organisations had grown in size, and now performed a great diversity of different types of work.  It was the age of the bureaucracy.  It was no longer possible for a small group of clerks to capture all incoming and outgoing communications within a series of books.   Files were a more flexible record keeping format, a file could be created whenever a piece of work started and closed when it finished

Was there a single organisation, operating in the second half of the twentieth century, that did not create hard copy files to record their work?

Records in our century

In the 21st century there is no such ubiquitous method of record keeping.

Records are still being kept.  But they are all over the place.   When you work with an organisation now you find different teams making different choices about how to record their work, and some teams not making a choice but leaving it down to individuals.   There is a safety net of sorts in the shape of the e-mail inbox, the place of first and last resort to find information in the enterprise.

The range of different ways in which people choose to keep records is dazzling.   I recently visited a compliance team in an organisation.   The team were conscientious, for each area of their work they had made a rational choice of how they keep their records:

• they kept records of their audit work as shared Outlook folders into which they dragged relevant incoming and outgoing e-mails
• they kept records of their policy work, and their approval work, as folders on the shared drive
• they kept records of their inspections on a dedicated line of business system
• they kept records of their project work as folders on the corporate document management system

The organisation concerned, like many others, is intending to roll out SharePoint over the coming year. That will provide yet more choices in terms of where and how teams store the records of their work.  They might use a SharePoint document library for each piece of work.  Or they might set up a SharePoint team site, or sub-site, for each piece of work.

It is significant that SharePoint is likely to come in alongside, rather than instead of, the systems that are already available to the team.

The implications of the lack of a standard method of keeping records for the records management profession

In such a situation we as a records management profession have three options:

• We can try to pursuade the 21st century to adopt a particular application or fomat for record keeping
• We can wait until the 21st century settles on a method of record keeping and in the meantime manage the vast array of different formats as best we can
• We can try to find a way of managing records that allows people to work (and store records) in any application that they choose, whilst at the same time allowing the organisation to control and manage those records over time

Each of these options has its pitfalls

The records management profession will not be able to dictate to the 21st century how it should keep records

The records management/archives profession has never been able, at any point in history, to dictate the way records were kept. In the 20th century records managers did not create the hard copy file.  It was the other way round – the hard copy file created records managers. Organisations were creating hard copy files anyway. Records management came into existence to solve the problem of how large 20th century organisations managed hard copy files at a corporate scale – to help them manage the thousands of files that were being created across a huge variety of different types of work.

In the first years of the 21st century the archives and records management profession tried to create a ubiquitous method of record keeping for the digital age.  It was to have been the ‘record folder’.  Note that we couldn’t take the word ‘file’ with us into the digital age because it had already been commandeered by Microsoft and others to denote a single digital object, such as a document produced in  MS Word.

Electronic records management systems (EDRMS) were specified by various national and pan national government bodies.   Record folders would be the equivalent of hard copy files.  They would be created when a piece of work started and closed when a piece of work finished.  They would be classified within a corporate fileplan.  They would receive the appropriate retention and access rule from the fileplan.  They would protect any document declared to that folder from amendment and from premature deletion.

Records folders have not become ubiquitous.  Some organisations use systems that support them, most do not. The phrase ‘record folder’ has not entered common parlance, it is a piece of jargon that any organisation implementing an electronic records management system has to establish, define, explain and promote.

The 21st century may never settle on a uniform way of keeping records

There is no guarantee that the 21st century will settle on a uniform way of keeping records. The competition between technology companies is driving constant development of new formats for the communication and storage of information. This competition is fuelled by the fact that the world wide web acts as a giant laboratory in which new collaboration/communication formats can be developed, tried and tested. When these formats are successful on the web they are inevitably brought into the enterprise after a time lag.

In the 20th century there was competition between companies supplying organisations with paper, filing stationery and filing cabinets. But none of those companies were trying to disrupt the way individuals communicated or stored information. No company was trying to come up with an alternative to the hard copy document, or an alternative to the practice of gathering hard copy documents together into files. They could not have been able to disrupt these practices even if they had wanted to.

In the 21st century there has been a constant stream of new formats and applications disrupting the way people record and communicate information. Documents are still important. But successive waves of new formats (e-mail, instant message, blogs, discussion boards, wikis, status updates) have come in alongside the document.  

Microsoft still dominate the communication and storage of information in the enterprise. The two dominant business formats are still the document and the e-mail. They are created, stored and managed with Microsoft tools: Word, Outlook and SharePoint. In 2009 Google launched a beta version of Google Wave, an innovative format for collaborating on outputs that explicitly tried to re-magine both e-mail and the document. Google wants to muscle into the enterprise, and would find it easier to do if they could move people onto a different format, a format invented by Google. We do not yet know whether or not Google Wave will be a success, so far it has not lived up to the hype. We do know that whenever a status quo is established there will always be another tech company trying to disrupt it.

This disruption is bound to continue: market dynamics will see to that. Even if the 21st century did settle on one method/application for keeping records, it would be disrupted within a decade.

We have no experience of implementing records systems that manage content stored in various different applications

We can characterise the situation at the end of the first decade of the 21st century as follows:

• Records can be, and are, generated in any application and any format that is generally available to knowledge workers
• More and more different applications and formats are available to 21st century knowledge workers
• More and more different formats and applications will continue to be made available
• The attempt to persuade/ cajole/ force colleagues to save all content needed as records into one application (the electronic records management system) looks doomed to only partial success

Lets make three assumptions

• organisations will continue to exist
• organisations will continue to want to have a corporate method of managing records
• organisations will continue to look to vendors of electronic content management (ECM) systems for solutions to the problem of managing records

If you can’t predict what format or what applications people are going to use within your organisation then your only option is to have a system that will manage records produced in any format, in any application.

The challenge for ECM vendors is :

• Make us a tool that can manage records kept in any format, in any application.  Convince us that it could be used to capture and manage records created in formats and applications that have not been invented yet

The challenge for us as a records management profession is even harder:

• If and when vendors did provide us with such a tool, how would we deploy it?

There is a vendor term for this model: ‘manage-in-place’. Vendors describe these systems as enabling you to hold a fileplan and retention rules within a manage-in-place Enterprise Content Management System (ECM), but apply them to content held elsewhere. For example you could apply a retention rule to content held in shared drives.

The rise of SharePoint has pushed ECM vendors out of the collaboration space, and sent them scrambling to find a new unique selling proposition. The ‘manage in place’ model seems to provide ECM vendors with a way of staying relevant to document and records management within the enterprise. Their new USP of the ECM is – ‘we can help you govern the content that your users create in SharePoint, shared drives, line of business systems etc.’

However there are a great many unanswered questions about the model. Even if the functionality is there, no-one knows how organisations could make it work. There are no case studies out there that I have heard of, and no guidance notes from any national archive or professional society.

Here are some questions about the implementation of a ‘manage-in-place’ records management system:

• Would all colleagues interact with it? Or would it sit in the background, used only by administrators and records managers?
• How would colleagues activate it when they wanted something to be captured as a record? For example lets imagine a team are using a shared drive to keep records of the audits that they carry out. They want those records to be protected and governed by the ‘manage in place’ ECM system. How do they go about that making that happen?
• How would colleagues know something had been captured as a record?
• Do records stay in the original application, or do they move to the records management tool?

The main question records managers ask me about SharePoint is ‘do I need to plug an ECM system into the back of it to manage records?’. Explaining the weaknesses in records management in SharePoint 2007 and 2010 is not enough to persuade SharePoint customers to buy ECM systems to manage records. The price tag of ECM licences, and the added complexity of implementing an additional application are strong barriers to overcome. The attitude of people seems to be ‘we are getting SharePoint anyway, we will see how we go with it, and if we need to plug an ECM in to govern the content we will do so at a later date’.

If the ECM vendors seriously want to win market share for their ‘manage-in-place’ records systems then they need to get serious about promoting the concept, explaining the model and sparking a debate about it. They would also need to slash their prices so they get some take-up, some case studies, and some momentum.

In a recent episode of the MOSS Show podcast Hilton Giesenow interviewed two highly experienced SharePoint administrators, Todd Klindt and Shane Young. Todd and Shane described the options for upgrading from SharePoint 2007 to 2010, and why they think the upgrade will be a boon for hardware manufacturers.

If you are running SharePoint 2007 you will only have two options for upgrading to SharePoint 2010:

• An in-place upgrade: SharePoint goes through all your content databases and converts them to SharePoint 2010. It does it all in one go, on the same hardware, with everything staying at the same url address as previously
• A database attach upgrade: you keep your SharePoint 2007 farm running, and set up a brand new server farm to run SharePoint 2010 alongside it. You move your content databases over one by one from the SharePoint 2007 farm to the SharePoint 2010 farm

At first sight the in-place upgrade appears the better option. It is quicker, simpler (because you don’t have the complexity of running 2007 and 2010 in parallel) and cheaper (because you can re-use your existing hardware).

However Todd and Shane said that their experiences of running in-place upgrades from SharePoint 2003 to 2007 were so bad that they are wary of trusting it this time around. The in-place option gives you no control over the upgrade process. You have to let the whole upgrade take place over the whole farm. If the upgrade has failed all you can do is revert completely back to SharePoint 2007, diagnose what caused the problem, fix that problem and try the whole thing all over again.

Todd’s experience of running in-place upgrades from SharePoint 2003 to 2007 was that diagnosing the problem that caused your in-place upgrade to fail was like ‘shooting fish in a barrel’ . The problem could have come from anywhere on any content database anywhere on the farm, and the error messages that SharePoint provided were of little or no help.

Todd and Shane will instead be recommending that their clients run a database attach upgrade. They admit that this is a more expensive, time consuming and complex option than the in-place upgrade. It requires you to buy a complete set of new hardware for SharePoint 2010 to run on. It results in have what Todd called ‘URL craziness” as you move the content databases over one by one to run on the SharePoint 2010 farm, whilst still running the remaining databases on SharePoint 2007 from the 2007 farm.

However the advantage of the database attach upgrade is that you have more control over the whole process. You isolate each content database and move them over one by one. If the upgrade to one database fails it doesn’t mean that you have to roll back the whole of the rest of the upgrade. And it is much easier to diagnose problems because you know which content database the problem has occurred in.

Todd and Shane said that if they were a hardware manufacturer like HP or Dell they would be rubbing their hands at the thought of all those upgrades from SharePoint 2007 to 2010, and all the extra hardware that organisations will be buying.

Organisations today are using two conflicting working definitions of a record. Neither definition is adequate on its own (which is why the two conflicting definitions co-exist with each other!).

The first of the two working definitions states that ‘records’ are those documents/communications that are needed as evidence of a piece of work. This definition comes from records management theory. Records management practice aims to capture these documents in a file/ records folder dedicated to each particular piece of work. The file/ records folder exists in a records system, which could be anything from a shared network drive, to a SharePoint team site, an electronic records management system, a line of business document management system, or a traditional filing cabinet.

Traditionally records management practice aims to ensure that the file/ records folder is the only information about a piece of work that survives over time. This translates into advice to colleagues on handling documents and e-mails such as ‘if it is important put it on the file, it if it isn’t important get rid of it ‘

If a team is confident that the file/records folder tells the whole story of the piece of work, then they can be confident in disposing of all information about that work lying outside of the file. The converse is also true – if you have not got confidence that the files in your records system(s) serves as a sufficient story of your organisation’s activities, then you are not going to be confident that you can delete material from other information stores such as e-mail in-boxes, and shared drives.

However for most pieces of work in an organisation ‘the file’ in a designated record system can no longer be regarded as the single source of evidence on that piece of work. This is down to two factors:

• it is highly likely that other pieces of evidence about that piece of work persist outside of the file, and that these pieces of evidence are findable or discoverable
• it is highly unlikely that each colleague has placed all significant documents/communications arising from that piece of work onto the file

I am struggling to think of a single organisation I have worked with that routinely delete e-mails after a period of months, or routinely deletes material on shared drives that is a year or two years old – they simply have not got enough confidence in their ‘records’ to risk losing important material like that.

The second working definition defines ‘records’ as being any information, in whatever form, that is held by that organisation. In other words any piece of information that exists anywhere: on the organisations network, in its filing cabinets, on people’s desktops, in desk drawers. This definition stems from the legal obligations placed on organisations. These obligations vary from jurisdiction to jurisdiction, but include data protection, freedom of information, and e-discovery. Such obligations make no distinction between those documents/communicaions that an organisation intended to keep as records, and those that have accidentally survived.

This definition is not adequate from a practical point of view either. When an organisation receives a data protection subject access request, or a freedom of information request or an e-discovery demand, they have to pull out all the stops and trawl their network. They might ask swathes of the organisation to go through their e-mail and unearth any messages regarding that matter. They might do searches of shared drives and document management systems. They unearth a pile or a mountain of material, that they then spend days or weeks sifting through. That is a massive pain, that an organisation is willing to put up with because that is the law that they are working under. But this approach would be overkill for most internal uses of record, such as the following scenarios

• You have taken over the management of a contract that has run into problems. You want to understand what has been agreed and discussed with the contractor up to this point.
• You have been asked to revise the service level agreement you offer to your customers. You want to understand the thinking behind the clauses in the current agreement. You want to read the discussions/correspondence that took place the last time the SLA was drafted
• A budgeted cost has changed between different versions of a business plan and you are trying to understand why

In each of the above scenarios you would be hoping that the colleagues involved in those pieces of work had kept a good file/records folder of that work, in which they had brought together all the relevant documents/communications relating to the work.

The dilemma for organisations is that they are caught between a rock and a hard place. They are not able to maintain good files across all areas of work. The concept of a file/records folder feels like an old-fashioned concept, a legacy from the hard-copy world of decades gone by. And yet organisations still need some means of relating together the records (however defined!) of a piece of work and managing them over time.

If the file/records folder was invented today, from scratch, with no history or legacy, what features would it have?