Thinking Records

The noughties was a decade of two halves for electronic records management systems in the UK

The first half of the decade: 2000 to 2005

The first half of the decade was a boom period. The UK National Archives (then called the Public Record Office) issued an influential statement of requrements (TNA 2002) for electronic records management systems, with an accompanying evaluation programme. Similar evaluation programmes were set up in other countries. The requirements included the ability to:

• hold a hierarchical corporate records classification (fileplan)
• link retention rules and access rules to the fileplan
• allow folders and documents to inherit retention and access rules from the heading of the fileplan that they are saved under

The acronym EDRMS (electronic document and records management systems) was applied to systems that complied with statements of requirements such as TNA 2002. In the UK large organisations in the public sector and in regulated sectors of the economy would routinely specify TNA 2002 compliance when issuing a tender for a corporate scale document management system.

Like most booms it was overdone. Some UK central government departments rushed into implemementations, spurred on by a Modernising Government target (all UK government departments must create new records electronically by 2004), and by a desire to be ready for the coming into force in January 2005 of the Freedom of Information regimes for England and Wales, and for Scotland. Corporate Fileplans proved difficult for organisations to construct and unpopular for users to use.

Nevertheless EDRMS had established itself as an integral part of the enterprise content management (ECM) landscape. All the big ECM suite vendors (IBM, Open Text, ECM, Oracle etc.) now provide an EDRMS offering within their suites, alongside their web content management, portal and workflow offerings.

The second half of the decade – 2005 to 2009

In the second half of the decade EDRMS has struggled. Some of the factors behind the decline are specific to the UK, but the more important ones are global. The Modernising Government target deadline passed unnoticed, and the fear of Freedom of Information subsided. The National Archives withdrew their testing regime at the end of 2005 to make way for a European Union sponsored regime (MoReq2). It would be three years before the MoReq 2 specification would be published, and by that time EDRMS was facing a new threat in the shape of SharePoint 2007.

SharePoint 2007 came out towards the end of 2006. It had no facility to apply a corporate fileplan, and an obscure and unworkable method for applying retention rules (in the shape of the SharePoint Records Centre). SharePoint 2007 devolves power down to the team level, and hence did not hit the same user adoption problems that EDRMS implementations often ran up against. Microsoft cleverly/generously included client licences for SharePoint in the Enterprise Agreements that most organisations that buy Windows, Exchange and Office already have with them. This meant that many organisations wanting corporate wide document management systems did not issue an invitation to tender (ITT): instead they simply started using the SharePoint licences that they already had. This has had a ruinous effect upon electronic records management system evaluation programmes, such as MoReq2, whose influence is predicated upon buyers specifying compliance when they issue an ITT.

So here we are at the start of 2010. The acronym EDRMS has almost disappeared from our language. What are the factors that will shape the way that the electronic records management system market develops over the coming years?

Make or break for MoReq2

MoReq 2 came out at a bad time for records management standards and evaluation programmes. The double whammy of the global downturn and the rise in SharePoint meant that there was less money to be made out of electronic records management systems. In order to comply with MoReq2 all vendors, even those meeting existing standards such as TNA 2002, would need to make investments in their products to make their necessary changes. Most of the vendors decided not to make those investments and have not yet submitted their products for testing. The result is that at the time of writing only one vendor (Fabasoft) has acheived compliance with the standard. In November 2009 I heard Doug Miles of AIIM tell an AIIM trade members meeting that so long as there was only vendor on the list then buyers would be unlikely to specify MoReq 2 compliance on their Invitations to Tender, because if they did so they would end up with a shortlist of one supplier.

As a result of this situation MoReq2 will be overhauled during the early part of 2010, with a view to making it less onerous for vendors to adapt their products. Some functionality that is currently labelled as mandatory will be downgraded to optional.

Rory Staunton sits on the MoReq2 governance committee, and he has written an interesting article in the January 2010 Records Management Bulletin. He does not pull any punches in the article.
He says:

The market reality is that records management software as a component market has almost collapsed. The sales of records management software licences from vendors have been falling before and during this recession, and most vendors are chasing a non-existent goal they call eDiscovery. US style eDiscovery is almost non existent in Europe, and MoReq2 in its original version was a basket case

Electronic records management has a tough battle ahead of it. Staunton believes that the best chance of MoReq 2 having an influence is to pursuade regulators to adopt MoReq 2

The way to win a game, of course, is to write the rules, so clever national archives, vendorrs and large users should target regulators in all sectors and propose that they adopt and extend MoReq2 based approaches…..The real success of MoReq2 will be judged in five years time, depending on the number of regulators that embracee and extend it.’

Staunton is thinking or regulators in fields ‘from envirnoment to nuclear, healthcare to telcos, retail to finance’

Staunton went on to say:

‘there is not time to waste if the records management community are to ride the wave of impending business regulations that are sure to emerge from the current and widespread breakdown in financial sector regulation. In market terms, Europe needs some basic approach to securing business information. Europe needs a more productive approach than the knee-jerk, lawyer-led, e-mail centric eDiscovery systems that the North Americans are adopting, which are a retrospective friction on business, and do not enhance compliance as they cure, rather than prevent, breaches of process’

The coming of SharePoint 2010

John Newton (chairman and CTO of Alfresco, a competitor to SharePoint) paints a bleak picture of the effect of SharePoint on ECM vendors, in a blogpost outlining his predictions for 2010:

You have to hand it to Microsoft; they seem to have scared the bejesus out of the traditional ECM vendors. They have all seemed to rolled over and played dead in the wake of SharePoint 2007

And it could get worse before it gets better. SharePoint 2010 is coming along with ‘ ever tighter integration with office, better web support, new records management, and claims to better scalability and administration’

Nevertheless Newton believes that SharePoint still has vulnerabilities, arising from the fact that organisations adopting it are tied into the whole Microsoft stack :

”Microsoft has chosen not to address its fundamental architectural flaw of storing content in the database and it is still an exclusively Microsoft-centric platform. Forget whatever database, operating system, language, browser you have – you better get used to SQL-Server, .NET, Windows, IE and don’t forget Silverlight. If the traditional vendors can’t battle the crap out of that, then they deserve to lose. The next 12 months will be critical during the transition to SharePoint 2010”

ECM and SharePoint: competition or partnership?

The reaction of ECM vendors to SharePoint 2007 has gone through 3 phases:

• At first ECM vendors believed that the difficulties of governing information in SharePoint 2007 would mean that SharePoint would not greatly threaten their market share.
• When it became clear that SharePoint 2007 was shipping at a rapid rate the ECM vendors started developing ‘web parts’ that could be deployed within the SharePoint environment. The web parts enabled people to save docutments to the ECM, edit them and search for them, without leaving their SharePoint environment.
• More recently vendors have realised that a much deeper integration with SharePoint is required. The problem with using web parts for integration with an ECM was that SharePoint knew nothing of the documents. Clients wanted the content to be available to SharePoint so that they had the option of using SharePoint functionality (such as workflows) with the documents and other content that they created.

Most of the ECM vendors that I have spoken to recently are adopting an approach to integration with SharePoint, 2007 which sees content stored in the ECM system, but made available to SharePoint so that to all intents and purposes SharePoint views it as SharePoint content

The ECM vendors have a reasonable tale to tell. Their selling proposition is:

• that you can use the ECM to house your fileplan, and to apply it to any system that you connect it to (not just SharePoint)
• you can apply the fileplan to any type of content (not just documents, but also content such as discussion board posts, blog posts etc.), and any level of granularity (not just documents, but folders of documents, document libraries or entire team sites)
• that they provides a more efficient model for storing content. SharePoint stores both the document/content itself and the metadata about it, in a database (SQL Server). ECM systems store the document/content in a repository, with only the metadata stored in a database
• that although the content is stored outside of SharePoint/SQL server, the content is made available to SharePoint so whatever you wanted to do in SharePoint with that content you would still be able to do.

However the vendors are facing an uphill battle. The problems for them are that:

• Most of the ECM vendors seem to have settled on the same model. They are fighting for a relatively small market, and customers may find it hard to differentiate them from each other.
• they are reacting to SharePoint, rather than causing problems for Microsoft. It is one way traffic. There is nothing that any ECM vendor has done over the past few years that Microsoft has felt compelled to react to. Microsoft has nothing to fear from the ECM vendors in the collaboration space. (Note that things are slightly different in the web content management and portal spaces where SharePoint has had less impact on the market share of traditional ECM vendors).
• As Microsoft gradually improve information governance within SharePoint they will leave less and less space for ECM systems in the collaboration space.
• ECM vendors will be reluctant to rock the boat with Microsoft. Several ECM vendors have told me that Microsoft has been very helpful to them, sharing their roadmap and plans for 2010 well in advance so that the ECM vendors can plan how their products will integrate with it. But it means that in the collaboration space ECM vendors are becoming partners rather than competitors with Microsoft.

It is interesting to note that the only two vendors that are trying to take Microsoft on in the collaboration space (Alfresco and Google) are not traditional ECM vendors. Alfresco provides open source collaborative and document management software. Google provides Google Apps from the cloud, as a subscription service.

Northumbria University’s Eduserv funded research project into the usage of SharePoint within UK Higher Education Institutions (HEIs) is drawing to a conclusion, and we are currently drafting the final report. The report will be published on Eduserv’s website early in 2010. Whlle drafing the report we came up with the following thoughts on what the future holds for SharePoint in Higher Education:

Beginnings of the adoption of SharePoint 2010
SharePoint 2010 will become available in the first half of 2010. Most HEIs will wait until a service pack has been issued before they think about upgrading to it, so it wll be 2011 before SharePoint 2010 starts to have an impact. SharePoint 2010 will bring improvements to the social computing functionality of My Sites, with Facebook/Twitter style status updates, and with tagging and bookmarking. My Sites are significant in an HE context because they are the part of SharePoint that HEIs consider providing to students as well as staff. We have hitherto seen lackluster take up of My Sites in HE. Some HEIs implementing SharePoint 2007 have decided not to roll out My Sites at all, others have only provided them to staff, others have made them available to staff and students but decided not to actively promote them. We are likely to see increasing provision and take up of My Sites from those HEIs that move to SharePoint 2010.

Fuzzy boundary between SharePoint implementations and Virtual Learning Environments (VLEs)
There is no prospect, in the near future, of SharePoint challenging Blackboard’s leadership in the market for institutional VLEs for teaching and learning. Most HEIs now have both an institutional VLE, and a SharePoint implementation. Institutional VLEs are accustomed to battling against web hosted applications such as Facebook for the attention of staff and students. They now also face competition internally from SharePoint. We are currently finding that SharePoint is used at the margins of teaching and learning, filling in for areas that VLEs are weaker at. HEIs have reported SharePoint’s use for one-off courses and small scale courses; for pieces of work requring students to do group collaboration work, and for work that can not fit within the confines of one course. Schools or faculties that don’t like their institution’s proprietary VLE have long been able to grab an open source VLE (such as Moodle) and build their own VLE in that. Now we are hearing of some schools grabbing SharePoint and building a school specific VLE in SharePoint. However SharePoint has a long way to go before it is anything more than marginal to teaching and learning.

Increase in average size of SharePoint implementations
At the point of time in which the research was conducted (summer and autumn of 2009) many of the implementations we looked at were at an early stage. The boom in SharePoint came in 2008 and 2009, as HEIs started to pick up on SharePoint 2007. We will see the maturation of many implementations which are currently less than a year old. This is likely to bring with it some governance challenges (for example ‘SharePoint sprawl’) which are not apparent when implementations are smaller. It will also increase the percentage of staff and students in HE familiar with SharePoint as a working environment. One HEI told us that some of their academics, unaware that the University are about to deploy SharePoint, have been asking for SharePoint because they have been working with colleagues at other institutions who are using it.

Increasing competition from Google Apps for the collaboration space
SharePoint seems to have seen off the competition from other proprietary ECM vendors in the collaboration space (though it faces strong competition from both proprietary and open source systems in the web content management space and the portal space). It seems that the most likely form of competition in the collaboration space will come in the shape of Google Apps which offers significantly less functionality, but operates on a web hosted subscription model which may appeal to HEIs that want to avoid the complexities of the configuration and management of SharePoint

Formation of at least one Higher Education SharePoint User Group
It is suprising that no Higher Education SharePoint user group yet exists. The formation of one or more such groups would seem to be essential given the high level of take up in the sector, the complexity of the product, the customisation and configuration challenges it poses, and the range of uses it can be put to. Such a user group or groups could:

• support the sharing of knowledge across the sector
• provide the sector with a voice in relation to both Microsoft and to vendors within the ecosystem around SharePoint
• enable the sector to explore the implications of Microsoft’s increasing dominance within higher education, as domination of the collaboration space is added to its domination of operating systems, e-mail servers, and office productivity software.

In SharePoint 2007 the records management functionality was very poorly thought through and consequently organisations were dependent on the ecosystem around SharePoint for help in either getting the records management functionality to work or in providing an alternative set of records management functionality so that records were managed outside of SharePoint.

The big question for SharePoint 2010 is to what extent will Microsoft provide records management functionality within the product that is coherent and robust enough for people to use without resort to the ecosystem around it?

SharePoint 2010 doesn’t go into beta test until next month. I am writing this post on the strength of the reports by bloggers of Microsoft’s presentations to the SharePoint 2009 Conference in Las Vegas last week – I haven’t seen the product itself .

Records management in SharePoint 2007

Let us start by reminding ourselves of the key problems with records management in SharePoint 2007:

• to protect a document from deletion and amendment you needed to send it to a records library within a SharePoint records centre. In the records centre an administrator can set up records centre libraries to protect and apply retention rules to documents declared as records
• you had to do an awful lot of work behind the scenes in order for a user to be able to right click on a document and send it to an appropriate records centre library. You would need to have set up a records centre library for that type of record; defined a content type that type of document; and specified a routing rule for the content type so that documents were routed to the appropriate records centre library
• the records centre did not support a useable fileplan hierarchy. It did allow you to set up different records libraries inside the records centre, and each records centre could have a folder structure. However the folders themselves had no functionality. Records could not inherit metadata or rules from the folders. You could not route documents to a specific folder within a records library, only to the records library itself.

I know of no organisation that used the SharePoint 2007 records management functionality ‘out of the box’. The only book written about records management in SharePoint 2010 is a book about how to do custom development in order to adapt SharePoint for records management.

For a description of records management in SharePoint 2007 read these detailed blogposts from the Microsoft UK SharePoint team. Or read my evaluation of records management in SharePoint 2007.

The records management model in SharePoint 2010

The biggest claim made at Las Vegas for SharePoint 2010 is that it includes in-place records management functionality.

The use of the phrase ‘in-place records management’ is interesting because it is very close to the concept of Manage-in-place used by some of the big ECM system vendors (for example Autonomy/Meridio and Open Text) and by the ECM industry body AIIM.

Manage-in place as used by the ECM vendor community

Manage-in-place, as developed by the ECM community, involves:

• keeping your records management classification (fileplan), retention rules, and access rules in one records management system
• Setting up connectors from that records management system to the various environments in which colleagues collaborate (shared drives, SharePoint team sites, etc)
• Using the records management system, to lock down, classify and apply retention and access rules to documents and information resources within the different systems that you set up connectors into, without moving the documents from their native system.

Manage-in-place is an acknowledgement that the attempt to persuade colleagues to keep all those documents/resources needed as records within one electronic document and records management system was doomed to fail. It is still in its infancy, and I have yet to find any advice on manage-in-place coming from any non-vendor sources. For example there is no advice emerging from the UK National Archives, which was the source of so much good advice on the traditional, pre-SharePoint 2007 EDRM system. Nor is there yet an established flag ship case study of manage-in-place for the records management community in the UK or world wide to discuss and learn from.

In-Place records management in SharePoint 2010

The In-Place records management capability for SharePoint 2010 appears to be more restricted in scope than manage-in-place. It appears that ‘In-Place records management’ simply means that a user can declare a document as a record within a team site, and it can be protected from amendment and deletion without having to be moved to a records centre.

Here is how Corey Roth described the feature

I know a handful of people that would say Records Management in MOSS 2007 was pretty much unusable. I think SharePoint 2010 will change their minds. The main issues were that you had to ship documents to an external library to manage them and the interface for applying holds was not very user friendly. Activating the In Place Records Management feature [in SharePoint 2010] will add a new button, Declare as Record, to the ribbon of document libraries. Clicking this marks the item as a record and I believe puts it in a read only state.

For it to qualify as ‘manage in place’ SharePoint 2010 would need to give us the ability to do the following tasks to a document or information resource without moving it from a Team Site:

• allocate the record to a folder classified within a hierarchical fileplan
• allow the document to inherit retention rules and/or access permissions from the folder it is contained in or the fileplan path that the folder is classified by.

On the basis of the reports I am reading I am not convinced that SharePoint 2010 will allow those two things to happen. it seems to me that SharePoint 2010 will support a fileplan within the records centre, but not across team sites (I would be happy to be proved wrong!)

Microsoft provide a useful explanation of managing records in SharePoint 2010 here. But it does not clarify the question as to whether or not a document declared as a record in the team site is linked to the fileplan kept in the records centre.

Elements of records management functionality in SharePoint 2010

Fileplan

The SharePoint 2010 records centre will support a fileplan hierarchy. This is good news and might actually make the records centre useable. However if I am right that a document has to move to the record centre in order for it to be governed by the fileplan then this is a weakness. I would rather the system allowed the fileplan to be applied to the document or information resource in its original context (usually the team site in which it was created or posted).

Folders

Folders will be given functionality in SharePoint 2010- this is a major change in Microsoft’s stance. SharePoint 2007 gave no functionality to folders. Microsoft, together with most SharePoint consultants, argued that using columns of metadata was a better way of organising document libraries than folders.

Sadalit Van Buren went to SharePoint Governance session at the Las Vegas and reported that

one surprising piece of the content for me was this recommendation:
• In SharePoint 2007, the best practice was: use metadata, not folders.
• In SharePoint 2010: use folders, inherit metadata

Document identifiers

SharePoint 2010 will support unique identifiers for documents

Eduserv have published the literature review on the usage of SharePoint by Higher Education Institutions (HEIs) written by myself and Julie McLeod. The review is available from here (PDF)

Here is an extract from the conclusions of the review:

Given the widespread use of SharePoint in HEIs there has been surprisingly little debate about its effectiveness or otherwise in this sector. This contrasts with vibrant debates within HE about Virtual Learning Environments (VLEs), and providers such as Blackboard and Moodle. (See for example the various sessions about VLE’s and the future of VLE’s in the September 2009 Alt-C conference).

The closest things to a forum for discussing SharePoint in HEI are the SharePoint User Groups provided by some JISC Regional Support Centres for both FE and HE institutions. However, not every region has one and they have mainly concentrated on further education rather than HE. They meet infrequently and there has been little in the way of outputs from them.

Many HEIs have given details of their implementations on their websites. Some have provided case studies or allowed vendors to provide case studies. Some, for example the University of Coventry and the University of Oxford, have provided information about their future plans for SharePoint. But no evaluations of specific SharePoint implementations have been found.

The lack of widespread debate on SharePoint, in comparison to the debate on VLE’s in general and on particular VLE providers, may stem from the fact that:

• SharePoint’s functionality spans many different stakeholder groups within HE (webmasters, internal communications teams, knowledge managers, records managers, faculty administrators, learning technologists etc.). There are few, if any, forums that gather together this range of interested parties. In contrast, VLE’s are specific to the learning technology community and are fully debated and discussed by that community.
• SharePoint 2007 is a proprietary system. The closest equivalents to SharePoint 2007 are Enterprise Content Management systems (ECM). However, SharePoint 2007 has had far more uptake in HE than any other ECM system. There are key features of SharePoint 2007 that differentiate it from other ECM systems. Unlike other ECM systems SharePoint is already available to many HEIs as part of their Microsoft Campus. Unlike other ECM systems SharePoint is a platform that is capable of being extended by custom development or by (often) custom integration with other third party products from the vast ecosystem that surrounds SharePoint 2007.
• The rise of SharePoint has been relatively recent. SharePoint 2007 is vastly more powerful than its predecessor SharePoint 2003, and hence the last two years have seen a vastly increased take up of the product.

On Thursday this week I gave a presentation yesterday on ‘SharePoint in Higher Education’ to TFPL’s 4th SharePoint Summit Conference.  Here are the slides that I showed:

My Sites are the part of SharePoint 2007 that is most directly in competition with web based social network applications.   In terms of useability and dynamism they don’t stand up too well in comparison with the likes of Facebook, Twitter and Linked In,  but they do have an important role to play in a SharePoint implementation.   A key point of interest when SharePoint 2010 is unveiled next month will be the extent to which Microsoft attempts to provide a more intuitive social computing facility within SharePoint.

My Sites provide an individual with three key benefits:

• the ability to display information about themselves to colleagues, through a profile and, if they wish, a blog
• the ability to store documents in document libraries, and either keep them private or collaborate on them with colleagues
• the ability to function as a personalised start page

These are functions that the individual could in theory use a combination of a Linked In account, a Blogger account, a Google docs account and an iGoogle page for.  However Brett Young, in a very useful and reflective blogpost describing his organisations experience with My Sites, argues against comparing My Sites and web 2.0 applications:

It is very tempting to compare My Sites to Facebook. It’s not a good idea to do that. First, it builds an unrealistic expectation. Facebook is a Web 2.0 application designed from the ground up as a social networking platform. My Sites are built on MOSS and WSS. Facebook is relatively intuitive. Probably very few people feel the need for Facebook training. On the other hand, My Sites are about as easy (or hard) to use as any other Microsoft application. Frequently functions are hidden deep within complex menus. Seemingly simple functions, such as deleting a list or library throw people off. They invariably come away thinking a My Site is nothing like Facebook.

The second reason that comparisons to Facebook are problematic is that the feature sets don’t align. Sure there are some slight similarities. However, there is nothing in My Sites equivalent to status updates, the comment wall, or friend activity tracking. (No, the colleague tracker web part doesn’t even come close.) That’s fine. My Sites do a lot of cool business-related stuff that Facebook cannot do, like document management, approval workflows, and lists. So, the point isn’t that one is better than the other; it is that while there may be some loose similarities, they are two completely different tools, designed for different uses. It is better to market My Sites for what it is, a document-based, personal workspace with some basic social networking capabilities.

If colleagues do make use of their SharePoint 2007 My Site then it helps the whole ecosystem of a SharePoint implementation.

• it adds a human element – when an individual contributes a document to a library anywhere in SharePoint a link will be provided to his or her My Profile page in their My Site. This means that if I see a document that interests me I can immediately click on the link to the contributor’s My Site to find out more about them. How good the information is on the profile depends upon whether the individual is motivated to contribute to it (though organisations can usually pre-populate basic role and contact data for staff from other databases).
• It helps internal communications: My Sites are places to which the organisation can push messages relevant to that individual. The individual can use their My Site to manage their subscriptions to other information sources in SharePoint, and to keep tabs on the various team sites that they belong to or are interested in.

A My Site is a SharePoint ’site collection’ in its own right, which means that the individual has adminstration rights over it. They can add new pages, or new sub-sites. They can add whatever SharePoint web parts they wish to pages within their My Site. They can give trusted colleagues permissions to view and/or contribute to documents, document libraries, web parts, pages or sub-sites within their My Site.

The high degree of control that an individual has over their My Site has pros and cons.

• On the plus side it means that it gives the individual scope to experiment and customise and is a great place for them to learn about the functionality available in SharePoint, the way it works and what you can do with it.
• On the down side some organisations have found that underneath their My Site  individuals have created what are in effect team collaboration sites, visible only to the select group that the individual has invited in to that site, and evading any controls the organisations has put in place to govern and limit the creation of new team sites.  Organisations can mitigate this risk of this by imposing quota sizes on My Sites

Steve Gaitten has written an excellent critique of My Sites in SharePoint 2007, which is much fairer than its title ‘SharePoint My Sites Suck’ would suggest.

Gaitten’s comes up with three killer criticism of My Sites:

• Useability – Gaitten says My Sites are ‘too hard to personalise – if we’re asking users to choose, add and configure Web Parts to create a basic user profile, it’s just not going to work. Many employees can’t do it and most just don’t care enough to make the effort.’
• Lack of lifestreaming features – ‘Status is absent. The single most important feature driving the adoption of Twitter and Facebook is the status message. What are you doing right now?’
• ‘Lack of integration with existing social networks. Social networking is one of those things where you need a lot of people to play for it to work. On that basis alone, I’m skeptical that viable social networks can be formed outside of the largest organizations. You need thousands of users updating profiles, status messages and contributing to the network to keep it interesting. A company with even a few thousand employees may find that there just isn’t enough critical mass for an active social network to form’.

Gaitten gives the reason for the weakness of social computing within SharePoint as being the fact that the My Site feature must have been thought of and planned as early as 2005, when social computing was in its infancy. (SharePoint 2007 was released late 2006). This is a fundamental weakness of packaged software as opposed to applications hosted by the providor.  Think how many times Facebook has radically changed the look, feel and operation of its application since SharePoint 2007 was released. Even if SharePoint 2010 was to come up with a social computing facility that matched Linked in, Facebook, Twitter and iGoogle now, how will it keep up as those applications, and whatever succeeds them, continue developing?

Google Wave has variously been described by commentators as an e-mail killer, a Twitter killer and a SharePoint killer.

To understand the reasons why Google is launching such an ambitious application we need to understand the two battles that Google is fighting at the moment:

their battle to wrest control of computing inside the organisation away from their great rivals Microsoft

their battle to retain the relevance of their search engine, developed in a previous era of the web’s development,  to the real time web

Google’s battle with Microsoft for the enterprise

The competition between Microsoft and Google is different from any past battle between giant companies.    Pepsi versus Coca-Cola pitted two companies against each other who sold the same thing to the same market.  The competition between the two companies produced no lasting change in the nature of soft drinks or the way that we consume them.

Google versus Microsoft pits two companies offering very different applications, delivered in a very different way.    This competition promises to be very disruptive and to lead to profound changes in the way information is exchanged within and between organisations.

Microsoft gets its money from organisations.  It has excellent relationships with IT departments and it develops products for those IT departments to install on their servers and implement inside organisations.

Google gets its money from the the ecosystem of the world wide web.  It makes most of its money from selling advertising on its own sites and others.  Most of its products and services are aimed at individuals rather than organisations, are given away free to users and are hosted by Google itself.

The worrying thing for Microsoft is that we have reached a point in time where it is the needs of the ecosystem of the world wide web that is driving technological innovation,  not the needs of the enterprise.

Information management practices created by the web ecosystem are being translated into the enterprise, and it is one way traffic.   Organisations are trying to adopt wikis, blogs, micromessaging.  The web ecosystem is not trying to adopt classifications, metadata profiles, content types, etc.

Monopolies and disruption

The reason why the battle between Google and Microsoft will be so disruptive and so profound is that it is a battle between monopolies.   You can’t compete with a monopoly.  You can only shake the ground beneath the monopoly’s feet, and undermine the foundations on which the monopoly was built.
In order to get its foot into the enterprise market Google will have to entirely disrupt the basis on which people handle information in the organisation.

Microsoft has a  monopoly of the operating systems that organisation’s use (Windows), and of the e-mail and document creation tools that organisations use (Outlook and Office).  This  has enabled them to build what is fast becoming a monopoly of collaboration and document management within organisations (through SharePoint).

Google would not be able to compete with Microsoft for the hearts and minds of IT departments.   IT departments are full of people trained to install, customise, administer, upgrade and integrate Microsoft products.  The tie-in is too strong to break.

Google’s previous attempts to break into the enterprise

Google has tried a succession of disruptive tactics to get into the enterprise.   Google Wave is a significant step up in this campaign.

Google Docs was a cut down version of Microsoft Word.  Google Apps is a cut down version of Microsoft SharePoint.   Neither has seriously dented Microsoft’s market share.

Google was competing not on functionality, but on method of delivery.

Google Docs takes advantage of the fact that Google does not need people to buy their software, they just need people to use their services.  Google Docs is provided free to individuals.  It’s killer strength is that it enables individuals in different organisations to collaborate around a document.  A key weakness of Microsoft software, including SharePoint, is that it is very focused on helping people within an organisation to collaborate and does not do enough to make it easy for an individual to collaborate with people outside of their organisation.

Google Apps is Google’s attempt to build on Google Docs by combining it with e-mail, calendaring and collaborative sites to produce a product they can sell it into organisations.  It has been reasonably succesful, with some high profile clients (Telegraph Media Group, University of Westminster).  Its selling points are that

• it is hosted by Google over the web so the organisation doesn’t need to puchase and configure a server farm for it
• it is significantly less complex than SharePoint
• there is just one version of Google Apps so there is no need to go through the disruption entailed by the need to upgrade to the next version of the product.

It is as though Google’s solution to the problem of not being able to win over IT departments is to provide products that don’t need or make use of the skills that IT departments have.

So far organisations have seemed to prefer SharePoint to Google Apps.  SharePoint’s advantages over Google Apps include:

• the ability to develop applications within it
• the ability to suck in data from line of business systems and display them in SharePoint
• the whole ecosystem of plug-ins that vendors are developing for SharePoint to fill in specific gaps or weaknesses.

SharePoint 2007’s integration with Microsoft Office is creating a stranglehold.   The stranglehold will be extended with SharePoint 2010, with its tighter inegration with Office 10.  You can perform more and more SharePoint tasks from within Word, Excel and Outlook.  No other suppliers can compete with that.  Other ECMs (enterprise content management systems)  will manage information better, with better means of applying taxonomies and management rules.  But Microsoft provides the software that people use to create documents, spreadsheets and e-mails in the first place, and organisations will increasingly fear that chosing an alternative ECM will mean they miss out on their colleagues being able to use those tools almost seamlessly with their content management system.

What is Google Wave?

Google has recognised that it cannot offer anything to compete with SharePoint in terms of organising documents and e-mails produced with Microsoft Office and Outlook. Instead it is offering in Google Wave a tool to entice people to gravitate away from those methods of collaborating on and communicating information.

Google Wave collapses all the distinctions between wikis, documents, e-mail and chat.  It also straddles the border between the enterprise and the web, the private and the public.

Let us suppose I type some text into a Google Wave and invite you to join it.  You have the choice as to whether you treat it as:

an e-mail (you can reply to it, at a time of your choosing)

the start of an instant message conversation (you can reply straight away and start a dialogue)

a document (you can leave comments at the end of the text, or at any point within it)

a wiki (you can go into my text and make your own changes or additions to it)

I can invite new people to join the Wave at any point.  I can show them the entire wave or selected parts of it.   They can press a replay button to see a run through of how the Wave has got where it is now.  They can leave a comment/reply at any point of the thread.

I could embed a wave to my blog or website, and people can contribute to the wave in much the same way as they would post comments to a blogpost.

The real potential of wave for the enterprise lies with ‘robots’.  Google Wave is open source and has an open API.  Developers can write ‘robots’ that do things in, or with, Google Waves.   For example ‘robots’ could be developed to:

• bring data in from other systems into a Wave
• link a Wave to other systems
• ccapture a Wave into other information systems
• perform useful tasks within the Wave such as spelling or translation.

The range of applications developed for the iPhone transformed the usefulness of that device.  The range of robots developed for Google Wave will dictate whether or not Wave does manage to become establish itself as a useful business tool.

What are Google Wave’s chances of breaking into the enterprise?

Communication, by its nature, is viral.  If people want to communicate via Google Wave then it will come into the enterprise.  Google Wave will succeed if people on the web chose this as their preferred way of working with each other.  If that happens then you will find an ecosystem developing around it, just as an ecosystem developed around Twitter.  This ecosystem will include:

• client applications to make it easier for people to work with Wave
• gadgets and robots to enable people and organisations to do more with the ‘Waves’  they set up,
• services taking the source code and providing an alternative Wave hosting service to Google

Google has made Wave open source.  Organisations can choose to take the source code and host it themselves on their servers if they wish to.   Google don’t care whether people use Wave hosted by Google, or by an alternative providor, or by enterprises.  they would simply rather people used the Wave format than Microsoft Word, Twitter or Facebook.

There is a strong prospect that ‘robots’ will be developed to work inside Google Wave in such a way as to enable Waves to be integrated with the various information systems in use in the enterprise.  Some ‘robots’ will enable people to draw upon data held in organisational systems whilst using Wave.  Other ‘robots’ will enable the organisation to capture Waves into their own information systems.

Read this superb post from Openwetware in which a scientist speculates on how Google Wave could be used to improve the process of collaborating on writing scientific research papers, with the help of:

• a ‘robot’ that checks and records agreement between participants on copyright/creative commons licensing
• a ‘robot’ that checks and formatting scholarly references
• a ‘robot’ for registering the wave with the appropriate institutional repository for the University concerned.

The possibility is there for organsiations to  deploy robots that capture Waves and register them with whatever type of content management system they deploy (institutional repository/ document management system/ SharePoint, EDRM etc.)

Google’s battle to retain the relevance of its search engine

Web 2 hasn’t found its search engine yet.  It is still using Google, which was developed for the first version of the web.

Google’s search predominance is based on its ability to rank web pages in relation to any particular search query, in accordance with the authority that the rest of the web holds those pages, as evidenced by inbound hyperlinks.

The fundamental weakness of the Google search model is that it takes time for an article or blogpost to garner enough inbound links to give it the authority it needs to reach the top of the Google search rankings.  Google search is fabulous for topics that I don’t know much about.  But it is lousy for keeping in touch with topics that I do know about.  If I want to keep in touch with the latest developments in records management and SharePoint I am much better off following people, words or tags on services like Twitter, Friend Feed or Delicious. In these services new blogposts, tweets or comments on that topic are highlighted to me as soon as someone notices them.

Google’s dominance of search is grounded in the days days when the web consisted of static webpages. The growth of Twitter, Facebook and RSS readers moves the web away from the static web page model, to a situation where people experience the web as what Dave Winer calls rivers of news.  Twitter and Facebook provides a flowing river of news, updates, comment and chat.   My RSS reader provides a river of news.

This is disrupting the world of search.  Twitter in particular is a problem for Google.  It does not provide Google with good content for its search results, and it provides an alternative to Google for those looking for information.

A tweet is part of a river of information.  It is rare for one tweet to be informative enough for it to be worth you clicking on a search result to view it.  For that reason there are few inbound links to individual tweets and it is hard for Google to rank them.

Google can’t point to Twitter conversations, because conversations are not threaded. Twitter archives are not accessible beyond a few months back.  People looking for information from Twitter do not need to use Google’s search engine.  They can run a search within Twitter,  they can look at a Twitter hashtag, or simply pose their question to their Twitter followers with a Tweet.

Facebook is a much more closed system than Twitter.  Google’s search engine is unable to search most of the huge content present within Facebook.  Facebook’s architecture is based on only people in your network of friends being able to see what you post.  Twitter’s has a much more open model (where by default any tweet you post is viewable by anyone on the web) and this has enabled Twitter to gain massive publicity.  Celebrities and others are using Twitter to publicise themselves, their opinions and their lives.  In response Facebook is gradually and awkwardly opening up.  But it is no friend of Google.  Facebook chose Microsoft rather than Google as an investment partner. Microsoft has invested $240 million in Facebook.

Google’s search model is under threat from the real time web.  The search engine for the real time web will filter search results to reflect:

• my point in time (what are people saying ‘now’ about this?)
• my point in space (what are people near here saying about this?)
• my social circle (what are my friends saying about this?)
• my circle of influence (what are the people and sites on the web that I follow saying about this?)

In order to deliver that real time search,  a  search engine would need to know who I am, where I am, what I am interested in, who I am friends with and who I read.

Google’s strategy has been to turn its search page into a portal,  the front page for a whole suite of tools for individual to use.  Once I have logged in with my Google account,  I can use Google reader for my RSS feeds, Google mail for e-mail, Google docs for my word processing, i-Google as my start page, picassa as my photo album.  The  more I do with my Google acount the more Google knows about me,  and the more likely it is that Google will be able, at some future point, to offer a personalised search experience to me.

So far this strategy has had limited impact on Google’s actual search results.   Google’s  biggest fears are that the real time web will be captured by services that either offer scant useful content for their engine to discover (Twitter) or are closed to their Search engine (Facebook).

Google Wave is an attempt to shape the way that the real time web develops, in a way that does produce useful content for search engines, and which avoids falling under the domination of any one providor (unless it is Google itself).  Google Wave allows people to have real time conversations, in public or private places, and to publish those conversations to the web. It takes real-time one stage further by allowing people to see you actually type in your contribution to the conversation.

Google will host Google Wave but have made it open source and opened up APIs to it so that other hosters could either provide clients for it, or act as an alternative host for wave.

David Coursey has pointed out that it wouldn’t take much of a front end on Google Wave for people to use it as a replacement to something like Twitter or Facebook.  It would just need a client that allowed:

• me to subscribe to public waves started by people I am interested in
• other people to subscribe to public waves started by me
• me to track all replies/contributions to waves that I have started

Where is the war between Microsoft and Google taking us?

Google is fighting on a war on two fronts.  Microsoft is fighting a war on one.   Google is  battling for control of computing within organisations to add to its existing dominance of the web.   Microsoft is trying to retain and  entrench its dominance of organisations,  but  has little or no prospect of having its products taken up by the ecosystem of the web.

If Microsoft and SharePoint wins the battle for the enterprise, then organisational computing stays very seperate from practices on the web.   If Google and Google Wave wins the battle then organisational computing almost merges with the ecosystem of the web.

I went to the AIIM Roadshow at Wembley Stadium last Friday (June 5). The main draw was the advertised keynote. Steve Marsh, SharePoint product manager for Microsoft UK, would give a talk entitled

‘How SharePoint fits within the ECM spectrum’

I was intrigued by the title, it implies that SharePoint is nestled proudly and comfortably underneath the umbrella of ECM. The reality is that SharePoint 2007 has driven a coach and horses through the ECM market, and we are only just beginning to see the impact of this on vendors, on professional practice and on standards.

My main interest in the product manager’s talk was to hear the messages he would give to AIIM. AIIM represents both ECM vendors and individual professionals interested in ECM. It must feel very ambivalent towards SharePoint. Many of the big vendors, particularly those with full blown document and records management systems (EDRMS) that meet TNA 2002 and MoReq2 requirements, are suffering from SharePoint’s huge grab of market share.

ECM vendors now have to make their products relevant to the post SharePoint 2007 world. Vast swathes of the document management functionality of their products are duplicated by SharePoint. Their unique selling proposition is now their robust information governance features, including their capacity to hold and apply fileplans and retention rules.

Two very recent developments pose risks for the big ECM vendors:

• There are rumours that SharePoint 2010 will have better records management features. This is a risk even if these features do not transpire in any particularly useful form (I am thinking of the much-hyped but hardly-used SharePoint records centre). The mere hint that SharePoint 2010 will make a reasonable hash of applying fileplans and retention rules will lead organisations to postpone buying decisions.
• New products are coming onto the market, that are not full blown document management systems, but which simply try to plug the governance gaps in SharePoint 2007. These products claim to be able to apply a fileplan and retention rules in the SharePoint environment. They claim to be able to protect those documents, document libraries and team sites needed as records. It is possible that such products may be offered for a fraction of the price of the big TNA 2002/MoReq 2 compatible ECM systems.

Doug Miles speaks on the changing model of ECM

The first speaker was Doug Miles, head of AIIM Europe. He said you can’t speak about ECM without mentioning SharePoint, and rolled out some statistics from AIIM surveys:

• only 12% of organisations surveyed have succeeded in implementing ECM systems corporate wide.
• 50% of organisations surveyed are now using SharePoint and a further 12% have plans to use it.
• 30% of the organisations implementing SharePoint already have an ECM system in place within their organisation. Doug spoke about the fact that some of these organisations are experiencing conflict and confusion, with the scope of their SharePoint implementation cutting across existing ECM system.

Doug Miles contrasted two alternative models for ECM:

• ‘ECM central’ : the old model for ECM involved using one ECM system to hold, govern, classify and apply rules to all your organisation’s important content (records, documents, webpages, etc).
• ‘manage in place’: the new model for ECM allows organisations to accept the fact that they have many different repostories and systems. They use ECM as a management layer to govern, classify and apply retention rules to records in many different systems, by enabling the ECM to set up connectors to these systems.

Doug said that the key supporting standard for the ‘Manage in Place’ ECM model will be the CMIS standard, generated by the some of the big ECM players, including Microsoft, EMC, IBM, Open Text and the open source provider Alfresco. The CMIS standard for document management has been agreed but the records management standard has not yet been finalised. CMIS is a technical standard that allows different content repositories to inter-operate with each other. Doug predicted that CMIS will be hugely important over the coming years.

My thoughts on Doug’s talk are:

• The ‘ECM central’ model was extremely challenging for organisations to implement (as Doug’s stat that only 12% of organisations surveyed by AIIM have managed to implement an enterprise wide ECM system shows).
• The rise of SharePoint 2007 will accelerate the move away from the ‘ECM central’ and toward the ‘manage in place’ model.
• The ‘ECM central model’ had the backing of standards issued by governments and national archives (TNA 2002, MoReq2 etc.) which offered reassurance to buyers. The big marketing challenge for the ‘manage in place’ ECM model is that there is no equivalent standard for that space. (CMIS is an enabling technical standard issued by the vendors).

Duncan Williams on SharePoint 2010

After Doug Miles we had an announcement that the Steve Marsh, Microsoft UK’s product manager for SharePoint wasn’t able to attend. He was replaced by Duncan Williams of DeltaScheme (a Microsoft Gold Partner). He explained that Steve Marsh was busy at work and was getting married shortly. It didn’t seem a terribly strong excuse.

In his talk Duncan gave some statistics to demonstrate the massive growth of SharePoint and the huge revenue stream it is giving to Microsoft. He told us that Microsoft no longer allocated a marketing budget for SharePoint, they don’t need to market it. They have changed the way they are incentivising their sales staff – they now reward them for increasing the adoption and usage of SharePoint by existing clients rather than for winning new customers.

The reason Steve Marsh didn’t speak at the AIIM roadshow was because he doesn’t need to. IT departments are buying his product anyway.

I later found out that Steve had told AIIM several weeks before that he was unlikely to be able to attend but it was convenient for AIIM to keep his name on the advertised programme. (I have no complaints about that: Duncan Williams gave a very useful insight into Microsoft’s roadmap for SharePoint, he was more candid than a Microsoft speaker would have been, and I very much enjoyed the event. It simply shows how much AIIM and any other event organiser in this space needs to cover SharePoint to draw in the crowds).

Duncan Williams said that Microsoft’s strategy is to integrate Outlook, SharePoint and Office together more and more tightly. He thought that Microsoft had missed an opportunity to get the integration between SharePoint 2007 and Office 2007 really tight, but he expected them to get it right with Office 2010 and SharePoint 2010.

Duncan hadn’t seen the beta code for SharePoint 2010, but he had heard from Microsoft that it would offer ‘better support for hierarchical fileplans’. Duncan pointed out that as SharePoint 2007 provides no support for hierarchical fileplans it is probably better to say that it will provide ’some support’ for them. He also expected SharePoint 2010 to have improved social networking features.

What do you do if you have a successful EDRMS, and your IT department starts piloting SharePoint 2007?

When the keynotes were over I had a wonder around the exhibition.

I spoke to the information manager of one of that minority of organistations that has an enterprise wide EDRMS in place. It has been in operation for several years and covers all of the several thousand employees of the organisation. It houses their fileplan, applies their retention rules, and has enabled them to switch off access to shared drives. In other words, an EDRMS success story.

He told me that his IT department has recently bought SharePoint and is now piloting SharePoint team sites.

‘Don’t tell me’ I said ‘your IT department says they are only piloting team sites for collaboration, and that people won’t keep records in the team sites, even though the team sites contain document libraries with capability for version control and customised metadata’

‘Thats right’ he said ruefully

We talked about the options open to him: none of them were particularly attractive:

• He could chose to do nothing and watch SharePoint team sites get rolled out. This would provide an alternative place for people to store and manage documents, confuse staff, and undermine their EDRMS
• He could insist that SharePoint team sites have document libraries removed so that documents have to be saved into the EDRMS. This would neuter the SharePoint team sites and render them far less effective as a collaboration tool
• He could integrate his EDRMS into SharePoint, so that SharePoint document libraries are replaced by a web part leading to the EDRMS. Colleagues could save and view documents within the team site, but the documents would be stored and managed within the EDRMS. This option may become less attractive over time, if Microsoft integrate Office and Outlook more and more tightly with future versions of SharePoint.