Google Wave has variously been described by commentators as an e-mail killer, a Twitter killer and a SharePoint killer.

To understand the reasons why Google is launching such an ambitious application we need to understand the two battles that Google is fighting at the moment:

their battle to wrest control of computing inside the organisation away from their great rivals Microsoft

their battle to retain the relevance of their search engine, developed in a previous era of the web’s development,  to the real time web

Google’s battle with Microsoft for the enterprise

The competition between Microsoft and Google is different from any past battle between giant companies.    Pepsi versus Coca-Cola pitted two companies against each other who sold the same thing to the same market.  The competition between the two companies produced no lasting change in the nature of soft drinks or the way that we consume them.

Google versus Microsoft pits two companies offering very different applications, delivered in a very different way.    This competition promises to be very disruptive and to lead to profound changes in the way information is exchanged within and between organisations.

Microsoft gets its money from organisations.  It has excellent relationships with IT departments and it develops products for those IT departments to install on their servers and implement inside organisations.

Google gets its money from the the ecosystem of the world wide web.  It makes most of its money from selling advertising on its own sites and others.  Most of its products and services are aimed at individuals rather than organisations, are given away free to users and are hosted by Google itself.

The worrying thing for Microsoft is that we have reached a point in time where it is the needs of the ecosystem of the world wide web that is driving technological innovation,  not the needs of the enterprise.

Information management practices created by the web ecosystem are being translated into the enterprise, and it is one way traffic.   Organisations are trying to adopt wikis, blogs, micromessaging.  The web ecosystem is not trying to adopt classifications, metadata profiles, content types, etc.

Monopolies and disruption

The reason why the battle between Google and Microsoft will be so disruptive and so profound is that it is a battle between monopolies.   You can’t compete with a monopoly.  You can only shake the ground beneath the monopoly’s feet, and undermine the foundations on which the monopoly was built.
In order to get its foot into the enterprise market Google will have to entirely disrupt the basis on which people handle information in the organisation.

Microsoft has a  monopoly of the operating systems that organisation’s use (Windows), and of the e-mail and document creation tools that organisations use (Outlook and Office).  This  has enabled them to build what is fast becoming a monopoly of collaboration and document management within organisations (through SharePoint).

Google would not be able to compete with Microsoft for the hearts and minds of IT departments.   IT departments are full of people trained to install, customise, administer, upgrade and integrate Microsoft products.  The tie-in is too strong to break.

Google’s previous attempts to break into the enterprise

Google has tried a succession of disruptive tactics to get into the enterprise.   Google Wave is a significant step up in this campaign.

Google Docs was a cut down version of Microsoft Word.  Google Apps is a cut down version of Microsoft SharePoint.   Neither has seriously dented Microsoft’s market share.

Google was competing not on functionality, but on method of delivery.

Google Docs takes advantage of the fact that Google does not need people to buy their software, they just need people to use their services.  Google Docs is provided free to individuals.  It’s killer strength is that it enables individuals in different organisations to collaborate around a document.  A key weakness of Microsoft software, including SharePoint, is that it is very focused on helping people within an organisation to collaborate and does not do enough to make it easy for an individual to collaborate with people outside of their organisation.

Google Apps is Google’s attempt to build on Google Docs by combining it with e-mail, calendaring and collaborative sites to produce a product they can sell it into organisations.  It has been reasonably succesful, with some high profile clients (Telegraph Media Group, University of Westminster).  Its selling points are that

• it is hosted by Google over the web so the organisation doesn’t need to puchase and configure a server farm for it
• it is significantly less complex than SharePoint
• there is just one version of Google Apps so there is no need to go through the disruption entailed by the need to upgrade to the next version of the product.

It is as though Google’s solution to the problem of not being able to win over IT departments is to provide products that don’t need or make use of the skills that IT departments have.

So far organisations have seemed to prefer SharePoint to Google Apps.  SharePoint’s advantages over Google Apps include:

• the ability to develop applications within it
• the ability to suck in data from line of business systems and display them in SharePoint
• the whole ecosystem of plug-ins that vendors are developing for SharePoint to fill in specific gaps or weaknesses.

SharePoint 2007’s integration with Microsoft Office is creating a stranglehold.   The stranglehold will be extended with SharePoint 2010, with its tighter inegration with Office 10.  You can perform more and more SharePoint tasks from within Word, Excel and Outlook.  No other suppliers can compete with that.  Other ECMs (enterprise content management systems)  will manage information better, with better means of applying taxonomies and management rules.  But Microsoft provides the software that people use to create documents, spreadsheets and e-mails in the first place, and organisations will increasingly fear that chosing an alternative ECM will mean they miss out on their colleagues being able to use those tools almost seamlessly with their content management system.

What is Google Wave?

Google has recognised that it cannot offer anything to compete with SharePoint in terms of organising documents and e-mails produced with Microsoft Office and Outlook. Instead it is offering in Google Wave a tool to entice people to gravitate away from those methods of collaborating on and communicating information.

Google Wave collapses all the distinctions between wikis, documents, e-mail and chat.  It also straddles the border between the enterprise and the web, the private and the public.

Let us suppose I type some text into a Google Wave and invite you to join it.  You have the choice as to whether you treat it as:

an e-mail (you can reply to it, at a time of your choosing)

the start of an instant message conversation (you can reply straight away and start a dialogue)

a document (you can leave comments at the end of the text, or at any point within it)

a wiki (you can go into my text and make your own changes or additions to it)

I can invite new people to join the Wave at any point.  I can show them the entire wave or selected parts of it.   They can press a replay button to see a run through of how the Wave has got where it is now.  They can leave a comment/reply at any point of the thread.

I could embed a wave to my blog or website, and people can contribute to the wave in much the same way as they would post comments to a blogpost.

The real potential of wave for the enterprise lies with ‘robots’.  Google Wave is open source and has an open API.  Developers can write ‘robots’ that do things in, or with, Google Waves.   For example ‘robots’ could be developed to:

• bring data in from other systems into a Wave
• link a Wave to other systems
• ccapture a Wave into other information systems
• perform useful tasks within the Wave such as spelling or translation.

The range of applications developed for the iPhone transformed the usefulness of that device.  The range of robots developed for Google Wave will dictate whether or not Wave does manage to become establish itself as a useful business tool.

What are Google Wave’s chances of breaking into the enterprise?

Communication, by its nature, is viral.  If people want to communicate via Google Wave then it will come into the enterprise.  Google Wave will succeed if people on the web chose this as their preferred way of working with each other.  If that happens then you will find an ecosystem developing around it, just as an ecosystem developed around Twitter.  This ecosystem will include:

• client applications to make it easier for people to work with Wave
• gadgets and robots to enable people and organisations to do more with the ‘Waves’  they set up,
• services taking the source code and providing an alternative Wave hosting service to Google

Google has made Wave open source.  Organisations can choose to take the source code and host it themselves on their servers if they wish to.   Google don’t care whether people use Wave hosted by Google, or by an alternative providor, or by enterprises.  they would simply rather people used the Wave format than Microsoft Word, Twitter or Facebook.

There is a strong prospect that ‘robots’ will be developed to work inside Google Wave in such a way as to enable Waves to be integrated with the various information systems in use in the enterprise.  Some ‘robots’ will enable people to draw upon data held in organisational systems whilst using Wave.  Other ‘robots’ will enable the organisation to capture Waves into their own information systems.

Read this superb post from Openwetware in which a scientist speculates on how Google Wave could be used to improve the process of collaborating on writing scientific research papers, with the help of:

• a ‘robot’ that checks and records agreement between participants on copyright/creative commons licensing
• a ‘robot’ that checks and formatting scholarly references
• a ‘robot’ for registering the wave with the appropriate institutional repository for the University concerned.

The possibility is there for organsiations to  deploy robots that capture Waves and register them with whatever type of content management system they deploy (institutional repository/ document management system/ SharePoint, EDRM etc.)

Google’s battle to retain the relevance of its search engine

Web 2 hasn’t found its search engine yet.  It is still using Google, which was developed for the first version of the web.

Google’s search predominance is based on its ability to rank web pages in relation to any particular search query, in accordance with the authority that the rest of the web holds those pages, as evidenced by inbound hyperlinks.

The fundamental weakness of the Google search model is that it takes time for an article or blogpost to garner enough inbound links to give it the authority it needs to reach the top of the Google search rankings.  Google search is fabulous for topics that I don’t know much about.  But it is lousy for keeping in touch with topics that I do know about.  If I want to keep in touch with the latest developments in records management and SharePoint I am much better off following people, words or tags on services like Twitter, Friend Feed or Delicious. In these services new blogposts, tweets or comments on that topic are highlighted to me as soon as someone notices them.

Google’s dominance of search is grounded in the days days when the web consisted of static webpages. The growth of Twitter, Facebook and RSS readers moves the web away from the static web page model, to a situation where people experience the web as what Dave Winer calls rivers of news.  Twitter and Facebook provides a flowing river of news, updates, comment and chat.   My RSS reader provides a river of news.

This is disrupting the world of search.  Twitter in particular is a problem for Google.  It does not provide Google with good content for its search results, and it provides an alternative to Google for those looking for information.

A tweet is part of a river of information.  It is rare for one tweet to be informative enough for it to be worth you clicking on a search result to view it.  For that reason there are few inbound links to individual tweets and it is hard for Google to rank them.

Google can’t point to Twitter conversations, because conversations are not threaded. Twitter archives are not accessible beyond a few months back.  People looking for information from Twitter do not need to use Google’s search engine.  They can run a search within Twitter,  they can look at a Twitter hashtag, or simply pose their question to their Twitter followers with a Tweet.

Facebook is a much more closed system than Twitter.  Google’s search engine is unable to search most of the huge content present within Facebook.  Facebook’s architecture is based on only people in your network of friends being able to see what you post.  Twitter’s has a much more open model (where by default any tweet you post is viewable by anyone on the web) and this has enabled Twitter to gain massive publicity.  Celebrities and others are using Twitter to publicise themselves, their opinions and their lives.  In response Facebook is gradually and awkwardly opening up.  But it is no friend of Google.  Facebook chose Microsoft rather than Google as an investment partner. Microsoft has invested $240 million in Facebook.

Google’s search model is under threat from the real time web.  The search engine for the real time web will filter search results to reflect:

• my point in time (what are people saying ‘now’ about this?)
• my point in space (what are people near here saying about this?)
• my social circle (what are my friends saying about this?)
• my circle of influence (what are the people and sites on the web that I follow saying about this?)

In order to deliver that real time search,  a  search engine would need to know who I am, where I am, what I am interested in, who I am friends with and who I read.

Google’s strategy has been to turn its search page into a portal,  the front page for a whole suite of tools for individual to use.  Once I have logged in with my Google account,  I can use Google reader for my RSS feeds, Google mail for e-mail, Google docs for my word processing, i-Google as my start page, picassa as my photo album.  The  more I do with my Google acount the more Google knows about me,  and the more likely it is that Google will be able, at some future point, to offer a personalised search experience to me.

So far this strategy has had limited impact on Google’s actual search results.   Google’s  biggest fears are that the real time web will be captured by services that either offer scant useful content for their engine to discover (Twitter) or are closed to their Search engine (Facebook).

Google Wave is an attempt to shape the way that the real time web develops, in a way that does produce useful content for search engines, and which avoids falling under the domination of any one providor (unless it is Google itself).  Google Wave allows people to have real time conversations, in public or private places, and to publish those conversations to the web. It takes real-time one stage further by allowing people to see you actually type in your contribution to the conversation.

Google will host Google Wave but have made it open source and opened up APIs to it so that other hosters could either provide clients for it, or act as an alternative host for wave.

David Coursey has pointed out that it wouldn’t take much of a front end on Google Wave for people to use it as a replacement to something like Twitter or Facebook.  It would just need a client that allowed:

• me to subscribe to public waves started by people I am interested in
• other people to subscribe to public waves started by me
• me to track all replies/contributions to waves that I have started

Where is the war between Microsoft and Google taking us?

Google is fighting on a war on two fronts.  Microsoft is fighting a war on one.   Google is  battling for control of computing within organisations to add to its existing dominance of the web.   Microsoft is trying to retain and  entrench its dominance of organisations,  but  has little or no prospect of having its products taken up by the ecosystem of the web.

If Microsoft and SharePoint wins the battle for the enterprise, then organisational computing stays very seperate from practices on the web.   If Google and Google Wave wins the battle then organisational computing almost merges with the ecosystem of the web.

I went to the AIIM Roadshow at Wembley Stadium last Friday (June 5). The main draw was the advertised keynote. Steve Marsh, SharePoint product manager for Microsoft UK, would give a talk entitled

‘How SharePoint fits within the ECM spectrum’

I was intrigued by the title, it implies that SharePoint is nestled proudly and comfortably underneath the umbrella of ECM. The reality is that SharePoint 2007 has driven a coach and horses through the ECM market, and we are only just beginning to see the impact of this on vendors, on professional practice and on standards.

My main interest in the product manager’s talk was to hear the messages he would give to AIIM. AIIM represents both ECM vendors and individual professionals interested in ECM. It must feel very ambivalent towards SharePoint. Many of the big vendors, particularly those with full blown document and records management systems (EDRMS) that meet TNA 2002 and MoReq2 requirements, are suffering from SharePoint’s huge grab of market share.

ECM vendors now have to make their products relevant to the post SharePoint 2007 world. Vast swathes of the document management functionality of their products are duplicated by SharePoint. Their unique selling proposition is now their robust information governance features, including their capacity to hold and apply fileplans and retention rules.

Two very recent developments pose risks for the big ECM vendors:

• There are rumours that SharePoint 2010 will have better records management features. This is a risk even if these features do not transpire in any particularly useful form (I am thinking of the much-hyped but hardly-used SharePoint records centre). The mere hint that SharePoint 2010 will make a reasonable hash of applying fileplans and retention rules will lead organisations to postpone buying decisions.
• New products are coming onto the market, that are not full blown document management systems, but which simply try to plug the governance gaps in SharePoint 2007. These products claim to be able to apply a fileplan and retention rules in the SharePoint environment. They claim to be able to protect those documents, document libraries and team sites needed as records. It is possible that such products may be offered for a fraction of the price of the big TNA 2002/MoReq 2 compatible ECM systems.

Doug Miles speaks on the changing model of ECM

The first speaker was Doug Miles, head of AIIM Europe. He said you can’t speak about ECM without mentioning SharePoint, and rolled out some statistics from AIIM surveys:

• only 12% of organisations surveyed have succeeded in implementing ECM systems corporate wide.
• 50% of organisations surveyed are now using SharePoint and a further 12% have plans to use it.
• 30% of the organisations implementing SharePoint already have an ECM system in place within their organisation. Doug spoke about the fact that some of these organisations are experiencing conflict and confusion, with the scope of their SharePoint implementation cutting across existing ECM system.

Doug Miles contrasted two alternative models for ECM:

• ‘ECM central’ : the old model for ECM involved using one ECM system to hold, govern, classify and apply rules to all your organisation’s important content (records, documents, webpages, etc).
• ‘manage in place’: the new model for ECM allows organisations to accept the fact that they have many different repostories and systems. They use ECM as a management layer to govern, classify and apply retention rules to records in many different systems, by enabling the ECM to set up connectors to these systems.

Doug said that the key supporting standard for the ‘Manage in Place’ ECM model will be the CMIS standard, generated by the some of the big ECM players, including Microsoft, EMC, IBM, Open Text and the open source provider Alfresco. The CMIS standard for document management has been agreed but the records management standard has not yet been finalised. CMIS is a technical standard that allows different content repositories to inter-operate with each other. Doug predicted that CMIS will be hugely important over the coming years.

My thoughts on Doug’s talk are:

• The ‘ECM central’ model was extremely challenging for organisations to implement (as Doug’s stat that only 12% of organisations surveyed by AIIM have managed to implement an enterprise wide ECM system shows).
• The rise of SharePoint 2007 will accelerate the move away from the ‘ECM central’ and toward the ‘manage in place’ model.
• The ‘ECM central model’ had the backing of standards issued by governments and national archives (TNA 2002, MoReq2 etc.) which offered reassurance to buyers. The big marketing challenge for the ‘manage in place’ ECM model is that there is no equivalent standard for that space. (CMIS is an enabling technical standard issued by the vendors).

Duncan Williams on SharePoint 2010

After Doug Miles we had an announcement that the Steve Marsh, Microsoft UK’s product manager for SharePoint wasn’t able to attend. He was replaced by Duncan Williams of DeltaScheme (a Microsoft Gold Partner). He explained that Steve Marsh was busy at work and was getting married shortly. It didn’t seem a terribly strong excuse.

In his talk Duncan gave some statistics to demonstrate the massive growth of SharePoint and the huge revenue stream it is giving to Microsoft. He told us that Microsoft no longer allocated a marketing budget for SharePoint, they don’t need to market it. They have changed the way they are incentivising their sales staff – they now reward them for increasing the adoption and usage of SharePoint by existing clients rather than for winning new customers.

The reason Steve Marsh didn’t speak at the AIIM roadshow was because he doesn’t need to. IT departments are buying his product anyway.

I later found out that Steve had told AIIM several weeks before that he was unlikely to be able to attend but it was convenient for AIIM to keep his name on the advertised programme. (I have no complaints about that: Duncan Williams gave a very useful insight into Microsoft’s roadmap for SharePoint, he was more candid than a Microsoft speaker would have been, and I very much enjoyed the event. It simply shows how much AIIM and any other event organiser in this space needs to cover SharePoint to draw in the crowds).

Duncan Williams said that Microsoft’s strategy is to integrate Outlook, SharePoint and Office together more and more tightly. He thought that Microsoft had missed an opportunity to get the integration between SharePoint 2007 and Office 2007 really tight, but he expected them to get it right with Office 2010 and SharePoint 2010.

Duncan hadn’t seen the beta code for SharePoint 2010, but he had heard from Microsoft that it would offer ‘better support for hierarchical fileplans’. Duncan pointed out that as SharePoint 2007 provides no support for hierarchical fileplans it is probably better to say that it will provide ‘some support’ for them. He also expected SharePoint 2010 to have improved social networking features.

What do you do if you have a successful EDRMS, and your IT department starts piloting SharePoint 2007?

When the keynotes were over I had a wonder around the exhibition.

I spoke to the information manager of one of that minority of organistations that has an enterprise wide EDRMS in place. It has been in operation for several years and covers all of the several thousand employees of the organisation. It houses their fileplan, applies their retention rules, and has enabled them to switch off access to shared drives. In other words, an EDRMS success story.

He told me that his IT department has recently bought SharePoint and is now piloting SharePoint team sites.

‘Don’t tell me’ I said ‘your IT department says they are only piloting team sites for collaboration, and that people won’t keep records in the team sites, even though the team sites contain document libraries with capability for version control and customised metadata’

‘Thats right’ he said ruefully

We talked about the options open to him: none of them were particularly attractive:

• He could chose to do nothing and watch SharePoint team sites get rolled out. This would provide an alternative place for people to store and manage documents, confuse staff, and undermine their EDRMS
• He could insist that SharePoint team sites have document libraries removed so that documents have to be saved into the EDRMS. This would neuter the SharePoint team sites and render them far less effective as a collaboration tool
• He could integrate his EDRMS into SharePoint, so that SharePoint document libraries are replaced by a web part leading to the EDRMS. Colleagues could save and view documents within the team site, but the documents would be stored and managed within the EDRMS. This option may become less attractive over time, if Microsoft integrate Office and Outlook more and more tightly with future versions of SharePoint.

Last Monday I had the pleasure of sitting on a panel at the RMS conference to answer a set of questions on the current state of electronic records management.  

Paul Duller chaired and the panel consisted of Jon Garde, Marc Fresco, Leanne Bridges and myself .

We were lucky enough to have had a very relevant and challenging set of six questions submitted to us in advance via the RMS conference website.

What follows are my recollections of some of the points that we made.   

1. Is Moreq2 a useful standard for supporting electronic information management systems or has it come too late to make a difference?

Jon:   MoReq 2 took a long time to come to fruition, and that was frustrating.  It is not a significant advance upon previous standards (TNA 2002) and is if anything longer and more complex for vendors.   But there is great potential in MoReq 2.   MoReq2 isn’t finished with the publication of the specification.  It is an ongoing project, and once the governance framework and testing regimes are in place we will start to see value

Leanne:  At the Audit Commission we used MoReq1 in our procurement of an EDRMS system,  these standards are useful in giving organisations confidence they are procuring a system that can do the job.  

Me:   No electronic records management standard published in 2008 could hope to have the same influence as the first standard published by the Public Record Office (now the National Archives) in 1999.  Back then vendors were unsure of how to do electronic records management, and the standard was able to create the market for EDRMS.  Now the market has matured.  There is much broader understanding of electronic records management among vendors and practictioners alike.    EDRM has morphed into ECM,  which adds into the mix functions that are out of the scope of records management standards, such as collaboration,  web content management and basic social networking functionality.  And a big gorilla has entered the market in Microsoft with its SharePoint product.  Microsoft has been able to sell SharePoint to organisations looking for an ECM without baseing its records management model on standards like MoReq2.

Marc:   Its a fact that many EDRMS implementations have failed.  But it is also a fact that many other projects has succeeded.  The success or failure is ascribable at least as much to the project management and change management efforts around the technology, not the technology itself.  MoReq 2’s relevance is shown by the number of languages it has been translated into.

2. There is a general belief out with the RM community that MS SharePoint implementation is relatively simple and easy and that EDRMS implementation is hard and burdensome on users and their organisation. What evidence can the panelists provide from their experience that supports or is at odds with this belief?

Me:   I am not sure that ‘easy’ is the right word for SharePoint:  the simple old shared drive is much easier than SharePoint!   For teams the advantage they get from SharePoint is that they it gives them more choice on how they set up their information environment.  But choice brings with it some elements of complexity.     Within their SharePoint team sites teams are having to decide how many document libraries they want,  what they want them for, whether they want version control enabled, whether they want to organise their libraries by folders or content types.  When they start a new piece of work they need to decide whether they want a whole new sub site, or a new document library, or a new folder within a document library. 

For an organisation SharePoint is certainly far from easy to govern:  it takes considerable ongoing thought and effort to control the creation of new sites to prevent SharePoint sprawl, and the SharePoint records centre is far from straightforward to implement if you wish to use that for records management.

Marc:  SharePoint is easy and cheap to implement so long as you don’t want to implement it well.  If you want to implement it well  then you will have to give just as much thought to issues of how you want SharePoint to be structured, how you set up metadata and search, and  about governance and access permissions, as you would with an EDRMS implementation.  And if you want to manage records well all commentators are agreed you are going to have to plug in an EDRMS capability at the back of it anyway.

 

3. Is there a future for independent EDRM vendors now that Microsoft is targeting this market?

 

Me:  the next version of SharePoint will be out in the first half of next year and will be called SharePoint 2010.  Although no one has seen it yet, industry analysts are predicting that there won’t be significant improvements in the records management capabilities.  [see Kathleen Reidy and CMS Wire].  I agree with the analysts:  adding MoReq 2 records management functionality to SharePoint would be a major rewrite of the software and I can’t see Microsoft  undertaking it in the near future.    

Microsoft are under no competitive pressure to improve records management,   they have been able to sell perfectly well without it, and none of their real competitors (Google, Apple, Amazon cloud computing) are offering records management capabilities.    The EDRMS vendors recognise that they are not big enough to compete with Microsoft and are falling over  backwards to develop offerings that plug into the back of SharePoint rather than trying to take it on.  

Cloud computing is a threat to Microsoft because Amazon and Google (with their massive data centres) have got into that space first.  Microsoft are moving into the cloud computing market now, and have developed Azure:  an operating system similar to Windows that Microsoft would host from their datacentres rather than organisations having to install it on their servers.  But this means that much of the development effort in upgrading SharePoint in future will go into making sure it can run in the cloud on Azure as well as on Windows.

Jon:  Microsoft have realised that the recent  ISO standards for document formats (Open Document Format and Office XML) threatens the monopoly they had through Microsoft Office.  The move to standards based formats means that documents produced in MS Office are now in an XML format and hence competitors  can offer word processing packages (such as Google docs and Open Office) that can produce documents to the same format.

SharePoint enables Microsoft to keep Office going, because they have tightly integrated SharePoint with Office, and provided functionality to Microsoft Word, Excel and Powerpoint users that aren’t available to users of  the equivalent products of their competitors.  In fact I woudn’t be surprised to see in future years Microsoft binding SharePoint in with Windows, so that any organisation that bought Windows got SharePoint embedded in with it.

There will always be a role for the independent providers,  that is where innovation comes from.

 

4. In light of the Government White Paper on Open Source and Open Standards, how can organisations justify not using an Open Source solution for RM challenges?

Marc:  It is going to be very interesting to see the progress of open source systems over the next few years.   In EDRMS sphere the open source offerings are relatively new to the market compared to the proprietary systems,  but  Alfresco are an open source company that has advanced plans to go for MoReq2 compliance.

Jon:  As a programmer I love the idea of open source.  But if I was an organisation going open source I wouldn’t start with EDRMS.  I would start by looking at the whole stack that the organisation was using:  for example an organisation with a standard Microsoft stack (Microsoft Office running on Windows servers, with SQL as the main database) could think about whether it wanted to run  open source alternatives (such as Open Office, Linux servers,  MySQL database).   And then think about whether it wanted to go for an open source EDRMS to manage the information on that stack.  Doing it this way round also gives time for the  open source EDRMS offerings to develop and mature.

One motivation for the Government’s paper on open source could be a gambit to put price pressure on Microsoft:  it is the UK government saying to Microsoft ‘don’t raise your prices to much because we are willing to move to open source alternatives’.

Leanne:  Open source is great but it should be done on a pragmatic basis:  I know someone whose organisation is trying to go open source for all their information systems:  and is finding it difficult in some areas to find good open source solutions, particularly where the solution needed is very specialised.

Me:  It found it interesting listening to Jeremy Tuck speak at a conference about Islington Council’s implementation of the open source ECM system  Alfresco.   In order to consider the Open Source solution the Council to adapt their procurement  procedures.  The proprietary systems had salespeople who prepared costed bids.  Islington had to prepare a statement of what the costs and benefits would be of taking the Open Source code for Alfresco and implementing it as an ECM, and then compare this with the bids from the proprietary vendors.  

 

 5. What do the panelists see as the future role for the Records Manager in the world of electronic information management?

Leanne:  a consultant and advisor to the rest of the organisation

Jon:  a mentor to people in the organisation

Me:  If  services like Facebook and Twitter continue to grow at their current rate  then in a year or two’s time almost everyone of working age will be on one of those services.   Blocking access to these services is easy for organisations when staff are using devices their employer has bought for them, on the organisation’s premises.   But it is not so easy  when people bring their own computing devices to work (netbooks cost less than £200. iPhones and other smartphonest are spreading like wildfire) nor when staff work from home.  And when customers start to want to contact people in the organisation by Twitter or Facebook it is a brave organisation that will cut themselves off from their customers.  Will working conversations start to migrate away from e-mail on to systems like Twitter and Facebook? If so what do organisations do?  Do they rely on staff copying important communications into organisational systems?  Do they deploy systems to interoperate with these Web 2.0 services?

Records management is the weakest link in SharePoint 2007. This weakness will not deter organisations from adopting the product. SharePoint’s strengths in other areas, particularly the flexibility it provides teams in setting up their own collaborative environment, together with the strength of Microsoft’s marketing position, has ensured the product has seen phenomenal uptake since its launch in November 2006.

Microsoft have come up with an innovative records management model of their own for SharePoint. They have not attempted to follow the EDRMS model laid out in MoReq 2 or TNA 2002, where a system must have the capability to hold a corporate records classification (fileplan), which can be linked to a set of retention rules and a set of access rules. The EDRMS model gives strong corporate control as these system require teams creating new folders for new pieces of work to place them within the fileplan where they inherit the appropriate retention and access rules. The cost of that control however is the need for teams to interact with a large corporate hierarchical classification.

The SharePoint records management model involves giving teams document management capabilities within document libraries in team sites,  whilst providing a separate record centre facility for the protection of documents needed as records, and for the application of retention rules to them. This model has the advantage of avoiding imposing a corporate classification on teams, allowing teams to organise their team sites the way they wish.  However in practice implementation of the records centre has not been easy for those organisations that have tried it. They have found it difficult to find robust processes for selecting and routing documents needed as records from the team site to the right place within the records centre.

The way SharePoint records management works is closely tied in with the way that document works in the product. In this post we will look first at how document management works in SharePoint 2007, then at how retention rules work, then  at some of the specific challenges of setting up and administering the SharePoint records centre.

 

How document management works within SharePoint

In SharePoint teams collaborate on documents within document libraries, which exist within team sites.

Version control

When setting up a document library teams have the choice of whether or not to enable version control and check in/check out.

Metadata (SharePoint content types)

If a team wants specific metadata to be captured about documents within a document library they do so by setting up one or more SharePoint content types for that document library. The team can either create entirely new content types, or they can choose from content types that the organisation has already set up within their SharePoint site collection.

When a team sets up a content type they determine which columns (fields) of metadata should be captured about those documents. They can set up pick-lists for any column that requires a controlled vocabulary. However out of the box SharePoint only supports flat pick-lists, not hierarchical controlled vocabularies.

SharePoint does not automatically allocate a unique system identifier (number or code) to documents saved into the system. This is a weakness as a unique identifier is a simple but very precise and useful piece of metadata.

If a team wants to set up document templates,  then they can do so by linking the template to a content type.  This enables a colleague to create a new document from within a SharePoint document library.

Organising document libraries: folder structures versus content types 

A team can use a folder structure to organise a document library, in which case it functions much like a shared drive. Alternatively they can set up SharePoint content types to allow colleagues to sort or filter the library by the different columns of metadata for each document: much as you would an Excel spreadsheet.

SharePoint 2007 is written to favour content types rather than folders for organising document libraries.

Folders in SharePoint have no more functionality attached to them than folders in a shared drive. For example you can not link a retention rule to a folder in a SharePoint document library. 

Content types are pivotal to SharePoint 2007. Without content types each document library is a silo. Content types offer the possibility to break out of silos. Where an organisation has many different teams doing a similar type of work, they can define a content type for that type of work. If each different team sets up that content type within the document libraries that they use for that work, then the content type provides a bridge bringing all that work together.

Retention rules can be linked to content types. However the content type will not apply that retention rule to documents within the team site. The content type will only apply the retention rule to a document once the document has been copied across to a ‘records centre’ within SharePoint.

Managing content types

Content types poses change management challenges, governance challenges and practical challenges:

• For teams coming new to SharePoint from shared drives, content types are the main difference between a shared drive and a document library.  Some teams are enthusiastic about the possibilities they offer, but others are wary and prefer to have a straightforward folder structure which has the advantage of familiarity. Helping teams make an informed choice between the two options is an area records managers can add considerable value within SharePoint implementations.
• It is very hard to stop content types proliferating. Out of the box SharePoint content types can be set up locally by a team site owner, or defined across an entire site collection. The risk is that teams set up a document library, are unaware that an appropriate content type already exists, and therefore create a new one. To get the most out of SharePoint content types an organisation would think through what metadata columns and content types it needed at the very start of the implementation. The organisation would also put governance arrangements in place so that when teams need a content type for their document libraries they would be required to either choose an existing one or justify the need for setting up a new one.
• SharePoint is asking content types to do several different things: to determine what metadata is captured about documents, to link retention rules to documents, and to specify a document template for documents. Sometimes there is a tension between these three roles. 
• A content type only functions within the boundary of a site collection: if you have more than one site collection you will need to set up new content type for each site collection. (Read this post for an excellent description of the SharePoint site collection and the reasons why you may need more than one of them)

 

Alternative ways to apply retention rules within SharePoint 2007

Retention rules in SharePoint are called expiration policies. SharePoint can apply expiration policies to libraries: document libraries in team sites, or records libraries in the Records Centre.
You have two options. You can either:

• Link your expiration policy to document libraries within team sites OR
• Link your expiration policies to content types and route documents needed as records to the SharePoint records centre where they will inherit the retention rule of the content type

Linking expiration policies to document libraries

Linking your expiration policies to document libraries within SharePoint has the advantage that the policy is applied to documents in situ: you do not have to move them to a records centre.  However it does constrain teams to restrict document libraries to records sharing a common retention rule, as you can only apply one retention rule per document library.  Microsoft never intended document libraries to be used for storing records: they intended that documents needed as records would be sent to the records centre.  The expiration policy on the document library is merely to ensure that  documents not needed as records are disposed of promptly.  Document libraries do not protect documents against amendment or deletion.  

I have not yet heard of an organisation that has taken the route of attempting to store its records in team site document libraries and apply expiration (retention) policies to those document libraries.   The significant administrative overhead in implementing the records centre option may tempt some organisations to use this route.

Linking expiration policies to content types

Microsoft’s intended model for records management within SharePoint involves the use of content types to link retention rules to records and to route records to a records centre.  In the records centre documents are protected from unauthorised amendment and deletion and the retention rule is applied to them.  

The big disadvantage of applying expiration policies via content types is that SharePoint will only apply that expiration policy to those documents once they have been routed from documents libraries to the SharePoint records centre. This gives you the significant administrative overhead of ensuring that documents from that content type are moved to the right place in the records centre at the right time. 

 

Critical assessment of the SharePoint records centre

This is how Microsoft intended records management to work within SharePoint 2007 (out of the box, no customisation)

• Expiration policies (retention rules) are defined
• Content types are defined, and have expiration policies linked to them.
• Document libraries are set up within team sites, with content types enabled
• A records centre is set up within SharePoint
• A records routing table is set up for each content type (this tells SharePoint where to locate documents of that content type when they are routed to the records centre)
• Colleagues collaborate on documents within document libraries in team sites
• When colleagues wish to save a document as a record they right click on it, and choose the option ‘send to records centre’.
• When the option ‘send to records centre’ is chosen, SharePoint takes a copy of the document and sends the copy to the records centre. It routes the document to the records library within the records centre designated for that content type. In the records centre the document is protected from amendment or deletion for as long as the retention period specified for that content type

The fact that SharePoint forces us to move documents from the team site to the records centre in order to protect them and apply retention rules to them poses us the following major challenges:

• How do we ensure that all or most documents needed as records are routed to the records centre?
• How do we avoid losing all the context that the document had when it sat within the team site?
• How do we ensure that the document is routed to the right place in the records centre?
• What do we do when teams use methods other than documents to record their work?
• How do we ensure that the records centre becomes a useful resource, that repays the resources expended in setting it up and maintaining it?

Ensuring all or most documents needed as records are routed to the records centre

Routing documents to the records centre depends upon colleagues right clicking on the document as it sits in their team site document library, and selecting the option ‘send to records centre’. The likelihood is that many colleagues will omit to take this step. Sending a copy of the document to the records centre adds no appreciable value to the team that created the document, who already have access to it within their team site.

DEFRA have tried an innovative approach of asking colleagues to indicate when they save a document whether a document is is of high, medium or low importance when they save it within a team site document library.   DEFRA have customised SharePoint to automatically route across to the record centre any document defined as high or medium importance that has not been changed for six months.

Avoiding loss of context when the document moves from the team site to the records centre

When a document is saved to a document library within a team site it possesses some context. That context is provided by the other activity that is taking place within the team site: the discussion forum, calendar, announcements, wikis, blogs, any other document libraries etc. It is also provided by the name of the team site itself, and by the navigation pathway to the team site. When a document is routed away from the team site to the records centre it loses most of that context.

Out of the box when a document is routed from a team site to the records centre it carries with it the metadata about the document itself, and the name of the document library that it sits in.    DEFRA customised the routing of documents to the records centre in order to ensure that a document additionally brings with it details of the navigation pathway down to the team site.

Ensuring that documents are routed to the right place within the records centre

Microsoft envisages that for every content type you create you have a designated records centre library ready to receive records of that content type. When a document is sent to the SharePoint records centre it is routed to whichever records centre library that content type has been linked to in the records routing table. One records centre library may be set up to receive records of more than one content type.

If a colleague sends a document to the records centre that is not allocated to a content type, or which is allocated to a content type that does not have a routing table set up for it, then the document is simply sent to an ‘unclassified’ library within the records centre.

Challenges arise because content types are best suited for document types (invoices, reports, contracts etc), whereas  the record of any particular piece of work (a project, a programme etc.) usually contains a variety of different document types.

Don Lueders has written this useful blog post describing how a client of his customised SharePoint so that they could set up record centre libraries for different types of work (rather than different types of document). The work around involves getting the records centre to determine which library to send the document to based on a metadata field within the content type rather than the content type itself: the metadata field in question can be set to capture the name of the piece of work.

Capturing records when teams use methods other than documents to record their work

Within a SharePoint team site, documents are just one of the means a team can use to record their work. They may alternatively choose to record an aspect of a piece of work with a wiki, a blog, a discussion forum. Information recorded within these outputs cannot be routed to a records centre. The records centre is only geared up to accept documents.

The most serious example of this relates to the ‘Document workspace’. The document workspace is a facility that allows a team to create a collaboration area around one particular document. This is useful where a document requires the input of several people, and it allows colleagues to have a discussion around that document within the workspace, as well as manage different versions of the document. SharePoint allows you to save the actual document produced in the document workspace to a document library and then route it to the records centre. SharePoint does not enable you to roll up the discussions that took place within the document workspace and send them to the records centre with the document.

Ensuring that the records centre becomes a useful resource, that repays the resources expended in setting it up and maintaining it

Records exist to satisfy the need of different stakeholders to understand a particular work. When we think back 20 years the hard copy file in a traditional filing system was trusted because it was the tool that the people who carried out the work used whilst they were working. In SharePoint the working tool is the team site, and my prediction is that most stakeholders in a piece of work are likely to get a better impression of that work by looking at the team site, than by looking at the selection of documents siphoned off from the team site to go to the records centre. I do not anticipate heavy usage of the records centre as a reference tool. Colleagues will have gained familiarity with team sites from working in their own team site, but will have little or no familiarity with the records centre. If the SharePoint records centre is not used as a reference tool, then it will not justify the considerable administrative effort necessary to sustain it.

 

Conclusions

It is important for the future of records management that we come up with an electronic records management model that enables us to capture records from a team without shackling them with a corporate file plan. I applaud Microsoft’s attempts to come up with such a model, but I don’t think that the SharePoint records centre is there yet. As it stands at the moment it is too fiddly, requires too much customisation and too much administrative effort to sustain, and yields too little information retrieval benefit.

It is hard enough governing team sites and document libraries within SharePoint. The records centre option means that we are additionally having to ensure that a records centre library is set up for each area of work, that the appropriate content type is set up and enabled for each document library in each team site, and that the routing table for each content type sends documents from the document library to the appropriate records centre library.

My prediction is that the majority of organisations that deploy SharePoint 2007 will not deploy the records centre. Those organisations that acknowledge a serious records management need will, if funds allow, use a third party system to apply classification and retention rules to records. Those organisations that do not do not acknowledge a serious records management need will concentrate their efforts on governing SharePoint to prevent SharePoint sprawl, and will live without the ability to apply retention rules (much as organisations have limped on with shared drives despite its lack of records management functionality).

Note:  most of the links from this post point to Don Lueders’ excellent SharePoint records management blog:  if you are intending to deploy the SharePoint records centre it is well worth reading through the archives of Don’s blog for explanations of how it works and tips on customisations.

I went down to Dorchester last Friday to speak at the March 20 meeting of the South West Group of the  Records Management South West Group.

The meeting was about Enterprise Content Management (ECM)/ Electronic Document and Records Management systems (EDRMS), but also included some very lively debates about SharePoint.

At the meeting we heard about the innovative approaches two local authorities are taking to EDRMS/ECM.

The different approaches of Local Government and Central Government to EDRMS

Central Government sector set the lead for EDRMS in the UK. The main wave of central government EDRM implementations took place between  2001 and 2006.  Local authorities started implementing EDRM later than central government, and have done so in a different way.

The central government EDRMS model involved:

• rolling EDRMS out across the whole Government department or agency
• setting up  one corporate fileplan to cover all of their work
• asking all individuals and teams to create folders within the fileplan for each piece of work that they carry out.  

It would be hard for local authorities to follow that model.  County Councils are large organisations (most have over 10,000 staff) with a diverse range of functions.  A corporate EDRMS roll out to a big County Council would be difficult because of the size of the file plan necessary to cover all their work, and because of the the timescale needed to roll the EDRMS out to all staff. 

The challenge with fileplans is that you can only hope to define the first three levels corporately.  Beneath that you are getting into the nitty gritty of the work and need to spend  time with each team or department to map their specific work into the fileplan. Once you get past 3,000 staff within an organisation the timescale and the effort involved for a corporate EDRMS based around a corporate fileplan becomes almost impossible to manage.

Dorset County Council’s approach

David Reeve spoke to us about the EDRMS implementation (ShARE) at Dorset CC. Dorset started implementing their EDRM system in 2004. Instead of a straight division by division corporate roll out they have taken the approach of plugging the EDRMS in behind different line of business databases. These are databases that manage data about a particular area of work but that lack the capability of managing documents and records.

In the first phase Dorset plugged the EDRMS behind their adult social care database, their highways maintenance database and their property database. Social workers can access the case file for an adult directly from the adult care database, Highway engineers can access the case file for a piece of maintenance work from the highways database etc..

Corporate benefits have been derived because the case files from these different areas of work:

• are kept on a single corporate EDRMS
• are organised by the agreed Dorset County Council fileplan (the top two levels of which were agreed at the start of the project)
• can have retention rules applied to them

Dorset have avoided the main disadvantage of corporate fileplans (the fact that users are asked to use a large hierarchical classification covering the whole organisation to navigate to stuff). Users can instead access case files through their line of business databases.

But there is no getting round the problem of scaleability.  In this excellent paper about the project, David states that integrating the EDRM system with existing databases

‘is a complex and time consuming process.  If this process was done on a corporate scale the project would take many years to complete’.

In the same paper David wrote about the further challenge of rolling out the EDRMS to staff who do not have a line of business database, and where the EDRMS needs therefore to work with the simple shared drive. Again it is time and scale that is the problem:

‘It will take significant time and effort to develop the corporate file-plan and migrate information from the current, unstructured shared drives to ShARE. It is estimated that a team of 2 people working on the file-plan development will take 5 years to complete the rolling implementation
programme.’

Wiltshire County Council’s approach

Many organisations are facing a choice as to whether to go for the flexibility of SharePoint or the structure and governance capabilities offered by an EDRMS/ECM. Chris Tinsley described  Wiltshire County Council’s  ‘horses for courses’ approach. They give the TRIM EDRMS system to areas of the Council where there is strong compliance needs around the records; and SharePoint to departments where the information risks are less and the primary need is for collaboration.  Wiltshire are also hoping to implement an integrated version of SharePoint and TRIM shortly.

SharePoint debates

The debates about SharePoint were started when David Reeve showed a slide outlining his reservations about records management in SharePoint.  The slide sparked a stream of comments from the floor.  The majority of comments  expressed concern at the records management capabilities of the product, and at the ways in which organisations were rolling it out without paying sufficient atttention to the governance and information management issues it poses.  The criticism was not unanimous however, and a lively debate ensued.   I have outlined some of the implications SharePoint poses for records management here