Science Fiction writer Bruce Sterling has posted a transcript of his February 6 talk to Transmediale 10 Atemporality for the creative artist. The talk investigates the impact of the real time web on all those cultural activities that depend in some way on narrative – including the writing of history.

”History books are ink on paper. They are linear narratives with beginning and ends. They are stories created from archival documents and from other books. Network culture is not really into that. Network culture differs from literary culture in a great many ways. And step one is that the operating system is an unquestioned given. The first thing you do is go to the operating system, without even thinking of it as a conscious choice.

Then there is the colossally huge, searchable, public domain, which is now at your fingertips. There are methods to track where the eyeballs of the users are going. There are intellectual property problems in revenue, which interferes with scholarship as much as it aids it. There is a practice of ‘ragpicking’ with digital material – of loops, tracks, sampling. There are search engines, which are becoming major intellectual and public political actors. There is ‘collective intelligence’. Or, if you don’t want to dignify it with that term, you can just call it ‘internet meme ooze’. But its all over the place, just termite mounds of poorly organized and extremely potent knowledge, quantifiable, interchangeable data with newly networked relations. We cannot get rid of this stuff. It is our new burden, it is there as a fact on the ground, it is a fait accompli.

There are new asynchronous communication forms that are globalized and offshored, and there is the loss of a canon and a record. There is no single authoritative voice of history. Instead we get wildly empowered cranks, lunatics, and every kind of long-tail intellectual market appearing in network culture. Everything from brilliant insight to scurillous rumor.

This really changes the narrative, and the organized presentations of history in a way that history cannot recover from.”

What are the implications of this for record keeping? If the writing of history as a single narrative becomes impossible, then is the attempt to keep a single narrative of a piece of work impossible and/or unnecessary too? Should we still try to create a file (electronic or paper) for every piece of work, that brings together all the significant documents and communications arising from that work?

In the same talk Sterling describes a new way of looking at history:

history is a story. And to write down the story of the fourteenth century, to just ask yourself – “what happened in the fourteenth century?” — is a very different matter from asking the atemporal question: “What does Google do when I input the search term ‘fourteenth century?

This is reminiscent of the switch we are witnessing in record keeping. A decade ago organisations were trying to keep good records of everything they did, to be able to tell the story of everything they did, to keep an organised narrative. Now the aim is reduced to being able to respond to e-discovery/ Freedom of Infomation/ Data Protection Subject Access requests.

Our traditional concept of a record is of a stable information resource: a file, sitting in a record store or archive, or in an electronic records management system. A file that stays unaltered and inviolate as it moves through time, a narrative waiting to be read and interpreted by different waves of colleagues, auditors and historians.

The e-discovery/freedom of information/subject access request is to ‘show us everything your systems know about the dismissal of employee y’ or ‘show us everything your systems know about the contract with company x’. The narrative only comes together when the request comes in. The answer to the question will change over time, as the organisation’s network and systems change, as the tools it has to search them change, as things drop off the network and are added to it. In the same way as Google’s answer to a search query on ‘the fourteenth century’ will change over time, as its algorithms changes, as the internet changes.

Even if an organisation has been able to keep a good file for a piece of work the e-discovery/freedom of information/subject access requestor will want to go beyond that file. It will want the organisation to dredge up material kept outside of that file that the requestor might find useful, advantageous or interesting.

The implication for records management is that we are in a transition away from managing static files towards managing shifting networks of information.

Alan Pelz-Sharpe is an incisive Enterprise Content Management system analyst. He blogged today that Google’s aggressive use of its users contact data in the recent launch of Google Buzz indicates that Google is unsuitable for the enterprise, as they can not be trusted with enterprise data.

Alan’s statement is important in the context of the battle between Microsoft and Google for the provision of cloud based e-mail, calendaring and collaboration applications to the enterprise. The battle has only recent commenced but it is already developed into both a price war and a war of words.

• Microsoft’s offering is BPOS (Business Productivity Online Standard Suite). BPOS provides enterprises with the following applications, hosted in the cloud: Exchange Online, SharePoint Online, Office Live Meeting, and Office Communications Online
• Google’s offering is Google Apps which packages up Gmail for business, Google sites, Google Groups, Google Docs, Google Calendar and Google Video

Both of these products are immature: Google Apps has been on the market longer, but Google are new to enterprise computing. Microsoft have vast experience of enterprise computing but little experience of providing software-as-a-service (SaaS).

In terms of functionality BPOS, and in particular SharePoint, provides far more features than Google Apps. If levels of service and price were comparable you would expect Microsoft to win this war (though small enterprises may prefer the relative simplicity of Google Apps).

The fact that Alan Pelz-Sharpe sees trustworthiness as a key factor in judging a SaaS vendor shows how different the SaaS space is from the traditional packaged software space. In the packaged software space you did not have to decide whether you could trust the vendor with your data – they would never see it. You simply needed to be satisfied with the features that their software possessed, the financial stability of the vendor and their road map for their product.

In contrast SaaS providers will be judged on the supplier’s reliability, quality of service, security and trustworthiness, as much as on the features of the software. The SaaS market plays in many ways to Google’s strengths. Their success has been built on the resilience of their infrastructure. Their search engine is hardly ever down. They have always guarded the secrets of how they configure their data centres as zealously as their search algorithm, indicating that they regard running data centres as a core strategic competency.

Conversely the move to cloud computing negates two of Microsoft’s strengths:

• Their monopoly, through Windows, of operating systems in the enterprise is negated because organisations can run applications off rival operating systems based in the cloud, or subscribe to applications (like Google Apps) that run off rival operating systems (like Linux).
• The expertise that enterprise IT departments have built up over the years in the administration, upgrade and extension of Microsoft products gave them an edge against competitors. Organisations were reluctant to buy software that their IT staff were not familiar with. In the SaaS world applications are administered and upgraded by the provider, and the scope for developing and extending them is limited

Google and Microsoft are big, cash-rich companies, with plenty of freedom to manoeuvre, who act in what they see as their best commercial interests. I do not want to condone Google’s aggressive use of Gmail users’ contacts data, (it made me glad I do not use Google to maintain my contacts). However the commercial imperatives behind Google’s over aggressive launch of Buzz are not the same as Google’s drivers to enter the SaaS space, and I do not think we can necessarily assume that Google is less trustworthy with enterprise data than its competitors purely on the basis of their actions in the launch of Buzz.

The reasons behind Google’s aggressive launch of Buzz lie in the threat that Facebook poses to its monopoly of the web advertising market. Google’s monopoly of web advertising stems from their monopoly of search, and from the fact that search was until recently overwhelmingly the major route through which individuals were directed to websites.

The growth of Twitter and in particular of Facebook has altered that. More and more traffic on the web is directed by links from friends on Facebook or Twitter, and this is reducing the influence of Google search results. Steve Rubel recently reported that Facebook now drives more traffic to major websites than Google. In another post Rubel said that ‘Facebook is unstoppable, they aren’t just the next Google, they’re the next web’

Google recognises that in the medium term search will lose out to discovery through social networks. People are more and more using their Twitter network, and/or their Facebook network to bring them news, rather than going out and searching for it on Google. Google will only be able to retain its dominance of advertising if it can provide a discovery service that is both personalised to you as an individual, and social. It needs to be tailored to who you are, where you are, who you are interested in, and who your friends are. This is why Google is turning itself from a search company into a software company, it needs you to use its applications because that is how it will get to know you.

Contacts data was at the crux of the controversy over the Buzz launch, and contacts data is crucial to Google. The key advantage that Google’s Android phones have over Apple’s iPhone is that when you log on to your Google account on a Google phone it instantly synchronises your Google contacts and Google calendar to your phone. In contrast getting my contacts and calendar synched between my Mac and my iPhone is a service that Apple charges me £59 a year for (it is worth it to me, and I love the service, but I do not think the charge is sustainable in the face of Google providing a similar service for free).

Contacts data is important to people. They want to keep it in their control as they move from job to job, and device to device. Facebook and LinkedIn function in many ways as address books, but they do not tend to hold phone numbers and have no coverage of people that are not users of those applications. The perfect contacts book would be available when I use my phone and when I use my laptop. It would link with my e-mail, and it would have links to my contacts’ identities on applications such as Twitter, Facebook and LinkedIn.

Google are in a better place than their rivals to provide just such an integrated contact service:

• Apple are dominant in smartphones, and the iPad could give them dominance of the netbook/tablet space. Their contacts and calendar applications are more pleasant to use than Google’s. But they are weak in e-mail provision and do not provide a social network.
• Google have the smartphones, the e-mail service, and now with Buzz they are making a play for the social network space. When the Chrome OS netbook is launched they will rival Apple in the netbook/tablet space
• Facebook is by far the strongest in the social network space, but does not (yet) do e-mails or phones or any sort of device. There have been rumours about a Facebook phone and a Facebook e-mail service for some time and both would make sense.

Google’s land grab with Buzz was a bid to get into the social network space before Facebook’s extension outwards became unstoppable. They needed to get as many people as possible using Buzz as quickly as possible and that is why they launched it on users with little or no warning. They pushed Gmail users at breakneck speed through a series of questions without giving them any way of judging what the consequences of their choices were. Google used Gmail users’ contacts to suggest people for them to follow, and to suggest them to others. However the consequence of this aggressive strategy was that some people found that lists of people that they frequently had Gmail correspondence with were being displayed on their Google profile.

It was an aggressive move, but it is an aggressive market. Google has more to lose from standing still and seeing Facebook overtake it than it does by aggressively moving to occupy some of Facebook’s space.

Alan Pelz-Sharpe points out in his post that the enterprise market accounts for only 1% of Google’s revenue. He is right that the enterprise market is far less important to Google than it is to its competitors (for example Microsoft). However Google’s war with Facebook is not taking place in the enterprise. Facebook is not trying to compete for the enterprise (Microsoft holds a significant stake in Facebook and the two companies have not encroached on each other’s space). I do not believe that Google has anything to gain by acting as fast and loose with enterprise data as it did with customer contact data when it launched Buzz. For these reasons I am reluctant to read across from their actions in the launch of Buzz to make any conclusions as to the trustworthiness or otherwise of Google in relation to enterprise data

Paul Buchheit is the brains behind Gmail. He then left Google and founded Friend feed which was recently acquired by Facebook.

Last week Paul blogged his opinion on the design of devices and applications.

Buchheit’s post was written in response to criticisms of Apple’s forthcoming iPad. Critics have listed all the things that the iPad won’t do that they would expect a tablet computer or netbook to be able to do. For Buchheit these critics are missing the point:

What’s the right approach to new products? Pick three key attributes or features, get those things very, very right, and then forget about everything else. Those three attributes define the fundamental essence and value of the product — the rest is noise. For example, the original iPod was: 1) small enough to fit in your pocket, 2) had enough storage to hold many hours of music and 3) easy to sync with your Mac (most hardware companies can’t make software, so I bet the others got this wrong). That’s it — no wireless, no ability to edit playlists on the device, no support for Ogg — nothing but the essentials, well executed.

We took a similar approach when launching Gmail. It was fast, stored all of your email (back when 4MB quotas were the norm), and had an innovative interface based on conversations and search. The secondary and tertiary features were minimal or absent. There was no “rich text” composer. The original address book was implemented in two days and did almost nothing (the engineer doing the work originally wanted to spend five days on it, but I talked him down to two since I never use that feature anyway). Of course those other features can be added or improved later on (and Gmail has certainly improved a lot since launch), but if the basic product isn’t compelling, adding more features won’t save it.

By focusing on only a few core features in the first version, you are forced to find the true essence and value of the product. If your product needs “everything” in order to be good, then it’s probably not very innovative (though it might be a nice upgrade to an existing product). Put another way, if your product is great, it doesn’t need to be good.

At the end of the post Buchheit made a comment that helps explain why so many enterprise applications suffer useability problems:

Disclaimer: This advice probably only applies to consumer products (ones where the purchaser is also the user ..). For markets that have purchasing processes with long lists of feature requirements, you should probably just crank out as many features as possible and not waste time on simplicity or usability.

Enterprises still purchase their applications and devices through a long list of feature requirements. Microsoft are kings of enterprise computing because they provide so many features in their products- SharePoint can be made to do almost anything. It isn’t just Microsoft that operates like this. Every Enterprise Content Management system vendor plays the same game, favouring features over useablity and design values, because it is features that they are judged on.

This contrast between the design of devices and applications intended to appeal directly to consumers and those intended to win through enterprise procurement processes is coming to a head.   More and more aspects of enterprise computing are becoming markets which consumer-targeted devices/applications are competing for.

Three years ago the battle for the smartphone market was a straight head to head between Research in Motion’s Blackberry and Microsoft’s Windows Mobile. Both assumed the smartphone market would go to the supplier that won the enterprise market,  and both crammed their smartphone full of features. Then Apple bought out the iPhone, with less functionality,  appealing direct to consumers, and a design that so intuitive that they did not have to ship a manual with the phone.  Now the future of the smartphone is a battle between two consumer companies: Google and Apple.

These two companies will also be the ones fighting it out for the tablet/netbook market. This year will see the launch of Apple’s iPad, and of Google’s Chrome OS – a new operating system for netbooks. Both will be bought as consumer devices but used for work. Read this post from Pete Gilbert for a discussion of how the iPad will support work in ‘the third place’ – outside of both the home and the office, on the move, in cafes, on trains, airport lounges etc.

The trend of enterprise computing is to require less and less to be installed on the PC, with the logical end result being that your device will only need a browser. For over a decade Microsoft have had a very safe income stream from Microsoft Office, which always had to be installed on the client device. Now it has to compete with Google Docs, which not only doesn’t require anything installed on the device, but is also free. Their solution is Office Web Apps, which will allow people to view and edit documents held in SharePoint 2010’s without having MS Office installed on the device.

As soon as you reach the point where only a browser is required on the device then (in theory!) it doesn’t matter what device an employee uses (so long as it doesn’t compromise security). That will work to the advantage of both the enterprise and the individual: the enterprise because they will no longer need to worry about providing every individual with a PC (and configuring and maintaining them) – the individual because they can choose their device of preference.

Consumerisation is a trend that affects applications as much as devices.

The one thing that every web 2.0 application has in common is that all of them were written to be used without training and without helpdesk support. It is conceivable that enterprises may start to regard training and helpdesk support as unwanted costs of applications (much like cloud computing vendors are encouraging enterprises to see system and server administration as unwanted costs).

Last week I heard Roger James, Director of IT at the University of Westminster, describe to the UNICOM Records Management update conference how the University provided Google Apps to all its students and staff without having to provide any training. It has been in use for over a year and they have so far only received 150 help desk calls. Google Apps is an amalgam of applications such as GMail, Google calendar and Google Docs that were first launched as free consumer products on the web.

This Wednesday I heard Simon Wardley give a fascinating overview of cloud computing to the Records Management Update 2010 Conference organised by UNICOM. Simon is responsible for Cloud Computing Strategy for Canonical.

Simon told us that cloud computing is difficult to define because it is not a thing, it is a transition.

The transition is from:

• computing-as-a-product (the enterprise buys in hardware, operating systems and applications as products which they install, configure, deploy and maintain themselves)

towards:

• computing as a service (the enterprise subscribes to applications which are hosted in the cloud, deployed on operating systems that are hosted in the cloud, sitting on hardware that exists somewhere on the cloud)

The transition period will be problematic and disruptive for companies. They won’t know who to trust, or what standards they should be insisting on, or which suppliers will be viable over time, or how they should govern their information in the cloud. There will exist many different cloud models during the transition period. Enterprises won’t just flick a switch and move all their IT infrastructure, all their development and all their applications straight over to the cloud.

Cloud computing is not a new phenomenon. John McCarthy and Douglas Parkhill both (separately) came up with the concept of utility computing in the 1960s. The necessary technology (virtualisation, which allows a provider to run several different virtual servers on one peice of hardware) was described by Jerry Popek et al in the 1970s. But the crucial attitude change came more recently, with the writings of Paul Strassman and Nicholas Carr who argue that the majority of IT activities in an enterprise do not give the organisation any strategic advantage. In other words IT has become a commodity, well enough defined that no enterprise gets a strategic advantage from doing it better or differently than their competitors.

Simon told us that some organisations will lose out in the transition. One of the key risks is supplier lock in. To avoid supplier lock in you need to be able to get your data out of your existing supplier. You also need alternative suppliers to be both willing and able to accept your data in the form that you extracted it from the previous supplier. The cloud has not yet reached that level of maturity, it is still in a transition period and there is no agreed de facto standard for APIs (the application programming interfaces through which organisations access, amend and add to the data held by their cloud provider). Most cloud providers have set up their own APIs to their services. Simon showed us a long list of different cloud APIs. If you choose a supplier whose API is not compatible with any other vendor then changing supplier would be hampered by the cost of adapting their data so that it can be deposited through the different API.

Simon showed us the complexity of the cloud computing stack. with its three levels of infrastructure, platform and applications.

• Software-as-a-service vendors provide applications – tools such as Google Apps, SharePoint Online and Salesforce that end users can utilise
• Platform-as-a-service vendors provide programming language support and development tools to enable an enterprise to develop new applications and to deploy those applications [see this information week article for a survey of the platform-as-a-service market]
• Infrastructure-as-a-service vendors provide the’ bare metal’ hardware, plus an operating system and a virtualisation tool

Simopn anticipates that there will be cloud computing disasters and failures.
Cloud computing creates dependencies up and down this stack. For example you may trust your software-as-a-service provider to provide you with a stable, reliable application. They in turn may trust their platform-as-a-service provider. But if the infrastructure-as-a-Service provider at the bottom of the stack fails, this would have a knock-on effect up the stack causing both the platform-as-a-service and the software-as-a-service to fail. Simon drew parallels with the the catastrophic banking collapse of late 2008, which resulted from bankers being shielded from the true risks of their investment through the existence of different layers separating their investments from the properties on which those investments were secured.

However Simon does not anticipate that the risks, disruption and losses involved in the transition to cloud computing will prevent that transition happening. Organisations that continue to host their own IT may find themselves at the wrong end of a competitive gap between them and organisations that use cheaper, more efficient cloud computing services. This may mean that many organisations will have eventually no choice but to move to a cloud computing model, in some shape or form, for most non-strategic computing provision.

The noughties was a decade of two halves for electronic records management systems in the UK

The first half of the decade: 2000 to 2005

The first half of the decade was a boom period. The UK National Archives (then called the Public Record Office) issued an influential statement of requrements (TNA 2002) for electronic records management systems, with an accompanying evaluation programme. Similar evaluation programmes were set up in other countries. The requirements included the ability to:

• hold a hierarchical corporate records classification (fileplan)
• link retention rules and access rules to the fileplan
• allow folders and documents to inherit retention and access rules from the heading of the fileplan that they are saved under

The acronym EDRMS (electronic document and records management systems) was applied to systems that complied with statements of requirements such as TNA 2002. In the UK large organisations in the public sector and in regulated sectors of the economy would routinely specify TNA 2002 compliance when issuing a tender for a corporate scale document management system.

Like most booms it was overdone. Some UK central government departments rushed into implemementations, spurred on by a Modernising Government target (all UK government departments must create new records electronically by 2004), and by a desire to be ready for the coming into force in January 2005 of the Freedom of Information regimes for England and Wales, and for Scotland. Corporate Fileplans proved difficult for organisations to construct and unpopular for users to use.

Nevertheless EDRMS had established itself as an integral part of the enterprise content management (ECM) landscape. All the big ECM suite vendors (IBM, Open Text, ECM, Oracle etc.) now provide an EDRMS offering within their suites, alongside their web content management, portal and workflow offerings.

The second half of the decade – 2005 to 2009

In the second half of the decade EDRMS has struggled. Some of the factors behind the decline are specific to the UK, but the more important ones are global. The Modernising Government target deadline passed unnoticed, and the fear of Freedom of Information subsided. The National Archives withdrew their testing regime at the end of 2005 to make way for a European Union sponsored regime (MoReq2). It would be three years before the MoReq 2 specification would be published, and by that time EDRMS was facing a new threat in the shape of SharePoint 2007.

SharePoint 2007 came out towards the end of 2006. It had no facility to apply a corporate fileplan, and an obscure and unworkable method for applying retention rules (in the shape of the SharePoint Records Centre). SharePoint 2007 devolves power down to the team level, and hence did not hit the same user adoption problems that EDRMS implementations often ran up against. Microsoft cleverly/generously included client licences for SharePoint in the Enterprise Agreements that most organisations that buy Windows, Exchange and Office already have with them. This meant that many organisations wanting corporate wide document management systems did not issue an invitation to tender (ITT): instead they simply started using the SharePoint licences that they already had. This has had a ruinous effect upon electronic records management system evaluation programmes, such as MoReq2, whose influence is predicated upon buyers specifying compliance when they issue an ITT.

So here we are at the start of 2010. The acronym EDRMS has almost disappeared from our language. What are the factors that will shape the way that the electronic records management system market develops over the coming years?

Make or break for MoReq2

MoReq 2 came out at a bad time for records management standards and evaluation programmes. The double whammy of the global downturn and the rise in SharePoint meant that there was less money to be made out of electronic records management systems. In order to comply with MoReq2 all vendors, even those meeting existing standards such as TNA 2002, would need to make investments in their products to make their necessary changes. Most of the vendors decided not to make those investments and have not yet submitted their products for testing. The result is that at the time of writing only one vendor (Fabasoft) has acheived compliance with the standard. In November 2009 I heard Doug Miles of AIIM tell an AIIM trade members meeting that so long as there was only vendor on the list then buyers would be unlikely to specify MoReq 2 compliance on their Invitations to Tender, because if they did so they would end up with a shortlist of one supplier.

As a result of this situation MoReq2 will be overhauled during the early part of 2010, with a view to making it less onerous for vendors to adapt their products. Some functionality that is currently labelled as mandatory will be downgraded to optional.

Rory Staunton sits on the MoReq2 governance committee, and he has written an interesting article in the January 2010 Records Management Bulletin. He does not pull any punches in the article.
He says:

The market reality is that records management software as a component market has almost collapsed. The sales of records management software licences from vendors have been falling before and during this recession, and most vendors are chasing a non-existent goal they call eDiscovery. US style eDiscovery is almost non existent in Europe, and MoReq2 in its original version was a basket case

Electronic records management has a tough battle ahead of it. Staunton believes that the best chance of MoReq 2 having an influence is to pursuade regulators to adopt MoReq 2

The way to win a game, of course, is to write the rules, so clever national archives, vendorrs and large users should target regulators in all sectors and propose that they adopt and extend MoReq2 based approaches…..The real success of MoReq2 will be judged in five years time, depending on the number of regulators that embracee and extend it.’

Staunton is thinking or regulators in fields ‘from envirnoment to nuclear, healthcare to telcos, retail to finance’

Staunton went on to say:

‘there is not time to waste if the records management community are to ride the wave of impending business regulations that are sure to emerge from the current and widespread breakdown in financial sector regulation. In market terms, Europe needs some basic approach to securing business information. Europe needs a more productive approach than the knee-jerk, lawyer-led, e-mail centric eDiscovery systems that the North Americans are adopting, which are a retrospective friction on business, and do not enhance compliance as they cure, rather than prevent, breaches of process’

The coming of SharePoint 2010

John Newton (chairman and CTO of Alfresco, a competitor to SharePoint) paints a bleak picture of the effect of SharePoint on ECM vendors, in a blogpost outlining his predictions for 2010:

You have to hand it to Microsoft; they seem to have scared the bejesus out of the traditional ECM vendors. They have all seemed to rolled over and played dead in the wake of SharePoint 2007

And it could get worse before it gets better. SharePoint 2010 is coming along with ‘ ever tighter integration with office, better web support, new records management, and claims to better scalability and administration’

Nevertheless Newton believes that SharePoint still has vulnerabilities, arising from the fact that organisations adopting it are tied into the whole Microsoft stack :

”Microsoft has chosen not to address its fundamental architectural flaw of storing content in the database and it is still an exclusively Microsoft-centric platform. Forget whatever database, operating system, language, browser you have – you better get used to SQL-Server, .NET, Windows, IE and don’t forget Silverlight. If the traditional vendors can’t battle the crap out of that, then they deserve to lose. The next 12 months will be critical during the transition to SharePoint 2010”

ECM and SharePoint: competition or partnership?

The reaction of ECM vendors to SharePoint 2007 has gone through 3 phases:

• At first ECM vendors believed that the difficulties of governing information in SharePoint 2007 would mean that SharePoint would not greatly threaten their market share.
• When it became clear that SharePoint 2007 was shipping at a rapid rate the ECM vendors started developing ‘web parts’ that could be deployed within the SharePoint environment. The web parts enabled people to save docutments to the ECM, edit them and search for them, without leaving their SharePoint environment.
• More recently vendors have realised that a much deeper integration with SharePoint is required. The problem with using web parts for integration with an ECM was that SharePoint knew nothing of the documents. Clients wanted the content to be available to SharePoint so that they had the option of using SharePoint functionality (such as workflows) with the documents and other content that they created.

Most of the ECM vendors that I have spoken to recently are adopting an approach to integration with SharePoint, 2007 which sees content stored in the ECM system, but made available to SharePoint so that to all intents and purposes SharePoint views it as SharePoint content

The ECM vendors have a reasonable tale to tell. Their selling proposition is:

• that you can use the ECM to house your fileplan, and to apply it to any system that you connect it to (not just SharePoint)
• you can apply the fileplan to any type of content (not just documents, but also content such as discussion board posts, blog posts etc.), and any level of granularity (not just documents, but folders of documents, document libraries or entire team sites)
• that they provides a more efficient model for storing content. SharePoint stores both the document/content itself and the metadata about it, in a database (SQL Server). ECM systems store the document/content in a repository, with only the metadata stored in a database
• that although the content is stored outside of SharePoint/SQL server, the content is made available to SharePoint so whatever you wanted to do in SharePoint with that content you would still be able to do.

However the vendors are facing an uphill battle. The problems for them are that:

• Most of the ECM vendors seem to have settled on the same model. They are fighting for a relatively small market, and customers may find it hard to differentiate them from each other.
• they are reacting to SharePoint, rather than causing problems for Microsoft. It is one way traffic. There is nothing that any ECM vendor has done over the past few years that Microsoft has felt compelled to react to. Microsoft has nothing to fear from the ECM vendors in the collaboration space. (Note that things are slightly different in the web content management and portal spaces where SharePoint has had less impact on the market share of traditional ECM vendors).
• As Microsoft gradually improve information governance within SharePoint they will leave less and less space for ECM systems in the collaboration space.
• ECM vendors will be reluctant to rock the boat with Microsoft. Several ECM vendors have told me that Microsoft has been very helpful to them, sharing their roadmap and plans for 2010 well in advance so that the ECM vendors can plan how their products will integrate with it. But it means that in the collaboration space ECM vendors are becoming partners rather than competitors with Microsoft.

It is interesting to note that the only two vendors that are trying to take Microsoft on in the collaboration space (Alfresco and Google) are not traditional ECM vendors. Alfresco provides open source collaborative and document management software. Google provides Google Apps from the cloud, as a subscription service.

Northumbria University’s Eduserv funded research project into the usage of SharePoint within UK Higher Education Institutions (HEIs) is drawing to a conclusion, and we are currently drafting the final report. The report will be published on Eduserv’s website early in 2010. Whlle drafing the report we came up with the following thoughts on what the future holds for SharePoint in Higher Education:

Beginnings of the adoption of SharePoint 2010
SharePoint 2010 will become available in the first half of 2010. Most HEIs will wait until a service pack has been issued before they think about upgrading to it, so it wll be 2011 before SharePoint 2010 starts to have an impact. SharePoint 2010 will bring improvements to the social computing functionality of My Sites, with Facebook/Twitter style status updates, and with tagging and bookmarking. My Sites are significant in an HE context because they are the part of SharePoint that HEIs consider providing to students as well as staff. We have hitherto seen lackluster take up of My Sites in HE. Some HEIs implementing SharePoint 2007 have decided not to roll out My Sites at all, others have only provided them to staff, others have made them available to staff and students but decided not to actively promote them. We are likely to see increasing provision and take up of My Sites from those HEIs that move to SharePoint 2010.

Fuzzy boundary between SharePoint implementations and Virtual Learning Environments (VLEs)
There is no prospect, in the near future, of SharePoint challenging Blackboard’s leadership in the market for institutional VLEs for teaching and learning. Most HEIs now have both an institutional VLE, and a SharePoint implementation. Institutional VLEs are accustomed to battling against web hosted applications such as Facebook for the attention of staff and students. They now also face competition internally from SharePoint. We are currently finding that SharePoint is used at the margins of teaching and learning, filling in for areas that VLEs are weaker at. HEIs have reported SharePoint’s use for one-off courses and small scale courses; for pieces of work requring students to do group collaboration work, and for work that can not fit within the confines of one course. Schools or faculties that don’t like their institution’s proprietary VLE have long been able to grab an open source VLE (such as Moodle) and build their own VLE in that. Now we are hearing of some schools grabbing SharePoint and building a school specific VLE in SharePoint. However SharePoint has a long way to go before it is anything more than marginal to teaching and learning.

Increase in average size of SharePoint implementations
At the point of time in which the research was conducted (summer and autumn of 2009) many of the implementations we looked at were at an early stage. The boom in SharePoint came in 2008 and 2009, as HEIs started to pick up on SharePoint 2007. We will see the maturation of many implementations which are currently less than a year old. This is likely to bring with it some governance challenges (for example ‘SharePoint sprawl’) which are not apparent when implementations are smaller. It will also increase the percentage of staff and students in HE familiar with SharePoint as a working environment. One HEI told us that some of their academics, unaware that the University are about to deploy SharePoint, have been asking for SharePoint because they have been working with colleagues at other institutions who are using it.

Increasing competition from Google Apps for the collaboration space
SharePoint seems to have seen off the competition from other proprietary ECM vendors in the collaboration space (though it faces strong competition from both proprietary and open source systems in the web content management space and the portal space). It seems that the most likely form of competition in the collaboration space will come in the shape of Google Apps which offers significantly less functionality, but operates on a web hosted subscription model which may appeal to HEIs that want to avoid the complexities of the configuration and management of SharePoint

Formation of at least one Higher Education SharePoint User Group
It is suprising that no Higher Education SharePoint user group yet exists. The formation of one or more such groups would seem to be essential given the high level of take up in the sector, the complexity of the product, the customisation and configuration challenges it poses, and the range of uses it can be put to. Such a user group or groups could:

• support the sharing of knowledge across the sector
• provide the sector with a voice in relation to both Microsoft and to vendors within the ecosystem around SharePoint
• enable the sector to explore the implications of Microsoft’s increasing dominance within higher education, as domination of the collaboration space is added to its domination of operating systems, e-mail servers, and office productivity software.

In SharePoint 2007 the records management functionality was very poorly thought through and consequently organisations were dependent on the ecosystem around SharePoint for help in either getting the records management functionality to work or in providing an alternative set of records management functionality so that records were managed outside of SharePoint.

The big question for SharePoint 2010 is to what extent will Microsoft provide records management functionality within the product that is coherent and robust enough for people to use without resort to the ecosystem around it?

SharePoint 2010 doesn’t go into beta test until next month. I am writing this post on the strength of the reports by bloggers of Microsoft’s presentations to the SharePoint 2009 Conference in Las Vegas last week – I haven’t seen the product itself .

Records management in SharePoint 2007

Let us start by reminding ourselves of the key problems with records management in SharePoint 2007:

• to protect a document from deletion and amendment you needed to send it to a records library within a SharePoint records centre. In the records centre an administrator can set up records centre libraries to protect and apply retention rules to documents declared as records
• you had to do an awful lot of work behind the scenes in order for a user to be able to right click on a document and send it to an appropriate records centre library. You would need to have set up a records centre library for that type of record; defined a content type that type of document; and specified a routing rule for the content type so that documents were routed to the appropriate records centre library
• the records centre did not support a useable fileplan hierarchy. It did allow you to set up different records libraries inside the records centre, and each records centre could have a folder structure. However the folders themselves had no functionality. Records could not inherit metadata or rules from the folders. You could not route documents to a specific folder within a records library, only to the records library itself.

I know of no organisation that used the SharePoint 2007 records management functionality ‘out of the box’. The only book written about records management in SharePoint 2010 is a book about how to do custom development in order to adapt SharePoint for records management.

For a description of records management in SharePoint 2007 read these detailed blogposts from the Microsoft UK SharePoint team. Or read my evaluation of records management in SharePoint 2007.

The records management model in SharePoint 2010

The biggest claim made at Las Vegas for SharePoint 2010 is that it includes in-place records management functionality.

The use of the phrase ‘in-place records management’ is interesting because it is very close to the concept of Manage-in-place used by some of the big ECM system vendors (for example Autonomy/Meridio and Open Text) and by the ECM industry body AIIM.

Manage-in place as used by the ECM vendor community

Manage-in-place, as developed by the ECM community, involves:

• keeping your records management classification (fileplan), retention rules, and access rules in one records management system
• Setting up connectors from that records management system to the various environments in which colleagues collaborate (shared drives, SharePoint team sites, etc)
• Using the records management system, to lock down, classify and apply retention and access rules to documents and information resources within the different systems that you set up connectors into, without moving the documents from their native system.

Manage-in-place is an acknowledgement that the attempt to persuade colleagues to keep all those documents/resources needed as records within one electronic document and records management system was doomed to fail. It is still in its infancy, and I have yet to find any advice on manage-in-place coming from any non-vendor sources. For example there is no advice emerging from the UK National Archives, which was the source of so much good advice on the traditional, pre-SharePoint 2007 EDRM system. Nor is there yet an established flag ship case study of manage-in-place for the records management community in the UK or world wide to discuss and learn from.

In-Place records management in SharePoint 2010

The In-Place records management capability for SharePoint 2010 appears to be more restricted in scope than manage-in-place. It appears that ‘In-Place records management’ simply means that a user can declare a document as a record within a team site, and it can be protected from amendment and deletion without having to be moved to a records centre.

Here is how Corey Roth described the feature

I know a handful of people that would say Records Management in MOSS 2007 was pretty much unusable. I think SharePoint 2010 will change their minds. The main issues were that you had to ship documents to an external library to manage them and the interface for applying holds was not very user friendly. Activating the In Place Records Management feature [in SharePoint 2010] will add a new button, Declare as Record, to the ribbon of document libraries. Clicking this marks the item as a record and I believe puts it in a read only state.

For it to qualify as ‘manage in place’ SharePoint 2010 would need to give us the ability to do the following tasks to a document or information resource without moving it from a Team Site:

• allocate the record to a folder classified within a hierarchical fileplan
• allow the document to inherit retention rules and/or access permissions from the folder it is contained in or the fileplan path that the folder is classified by.

On the basis of the reports I am reading I am not convinced that SharePoint 2010 will allow those two things to happen. it seems to me that SharePoint 2010 will support a fileplan within the records centre, but not across team sites (I would be happy to be proved wrong!)

Microsoft provide a useful explanation of managing records in SharePoint 2010 here. But it does not clarify the question as to whether or not a document declared as a record in the team site is linked to the fileplan kept in the records centre.

Elements of records management functionality in SharePoint 2010

Fileplan

The SharePoint 2010 records centre will support a fileplan hierarchy. This is good news and might actually make the records centre useable. However if I am right that a document has to move to the record centre in order for it to be governed by the fileplan then this is a weakness. I would rather the system allowed the fileplan to be applied to the document or information resource in its original context (usually the team site in which it was created or posted).

Folders

Folders will be given functionality in SharePoint 2010- this is a major change in Microsoft’s stance. SharePoint 2007 gave no functionality to folders. Microsoft, together with most SharePoint consultants, argued that using columns of metadata was a better way of organising document libraries than folders.

Sadalit Van Buren went to SharePoint Governance session at the Las Vegas and reported that

one surprising piece of the content for me was this recommendation:
• In SharePoint 2007, the best practice was: use metadata, not folders.
• In SharePoint 2010: use folders, inherit metadata

Document identifiers

SharePoint 2010 will support unique identifiers for documents

Eduserv have published the literature review on the usage of SharePoint by Higher Education Institutions (HEIs) written by myself and Julie McLeod. The review is available from here (PDF)

Here is an extract from the conclusions of the review:

Given the widespread use of SharePoint in HEIs there has been surprisingly little debate about its effectiveness or otherwise in this sector. This contrasts with vibrant debates within HE about Virtual Learning Environments (VLEs), and providers such as Blackboard and Moodle. (See for example the various sessions about VLE’s and the future of VLE’s in the September 2009 Alt-C conference).

The closest things to a forum for discussing SharePoint in HEI are the SharePoint User Groups provided by some JISC Regional Support Centres for both FE and HE institutions. However, not every region has one and they have mainly concentrated on further education rather than HE. They meet infrequently and there has been little in the way of outputs from them.

Many HEIs have given details of their implementations on their websites. Some have provided case studies or allowed vendors to provide case studies. Some, for example the University of Coventry and the University of Oxford, have provided information about their future plans for SharePoint. But no evaluations of specific SharePoint implementations have been found.

The lack of widespread debate on SharePoint, in comparison to the debate on VLE’s in general and on particular VLE providers, may stem from the fact that:

• SharePoint’s functionality spans many different stakeholder groups within HE (webmasters, internal communications teams, knowledge managers, records managers, faculty administrators, learning technologists etc.). There are few, if any, forums that gather together this range of interested parties. In contrast, VLE’s are specific to the learning technology community and are fully debated and discussed by that community.
• SharePoint 2007 is a proprietary system. The closest equivalents to SharePoint 2007 are Enterprise Content Management systems (ECM). However, SharePoint 2007 has had far more uptake in HE than any other ECM system. There are key features of SharePoint 2007 that differentiate it from other ECM systems. Unlike other ECM systems SharePoint is already available to many HEIs as part of their Microsoft Campus. Unlike other ECM systems SharePoint is a platform that is capable of being extended by custom development or by (often) custom integration with other third party products from the vast ecosystem that surrounds SharePoint 2007.
• The rise of SharePoint has been relatively recent. SharePoint 2007 is vastly more powerful than its predecessor SharePoint 2003, and hence the last two years have seen a vastly increased take up of the product.

On Thursday this week I gave a presentation yesterday on ‘SharePoint in Higher Education’ to TFPL’s 4th SharePoint Summit Conference.  Here are the slides that I showed:

My Sites are the part of SharePoint 2007 that is most directly in competition with web based social network applications.   In terms of useability and dynamism they don’t stand up too well in comparison with the likes of Facebook, Twitter and Linked In,  but they do have an important role to play in a SharePoint implementation.   A key point of interest when SharePoint 2010 is unveiled next month will be the extent to which Microsoft attempts to provide a more intuitive social computing facility within SharePoint.

My Sites provide an individual with three key benefits:

• the ability to display information about themselves to colleagues, through a profile and, if they wish, a blog
• the ability to store documents in document libraries, and either keep them private or collaborate on them with colleagues
• the ability to function as a personalised start page

These are functions that the individual could in theory use a combination of a Linked In account, a Blogger account, a Google docs account and an iGoogle page for.  However Brett Young, in a very useful and reflective blogpost describing his organisations experience with My Sites, argues against comparing My Sites and web 2.0 applications:

It is very tempting to compare My Sites to Facebook. It’s not a good idea to do that. First, it builds an unrealistic expectation. Facebook is a Web 2.0 application designed from the ground up as a social networking platform. My Sites are built on MOSS and WSS. Facebook is relatively intuitive. Probably very few people feel the need for Facebook training. On the other hand, My Sites are about as easy (or hard) to use as any other Microsoft application. Frequently functions are hidden deep within complex menus. Seemingly simple functions, such as deleting a list or library throw people off. They invariably come away thinking a My Site is nothing like Facebook.

The second reason that comparisons to Facebook are problematic is that the feature sets don’t align. Sure there are some slight similarities. However, there is nothing in My Sites equivalent to status updates, the comment wall, or friend activity tracking. (No, the colleague tracker web part doesn’t even come close.) That’s fine. My Sites do a lot of cool business-related stuff that Facebook cannot do, like document management, approval workflows, and lists. So, the point isn’t that one is better than the other; it is that while there may be some loose similarities, they are two completely different tools, designed for different uses. It is better to market My Sites for what it is, a document-based, personal workspace with some basic social networking capabilities.

If colleagues do make use of their SharePoint 2007 My Site then it helps the whole ecosystem of a SharePoint implementation.

• it adds a human element – when an individual contributes a document to a library anywhere in SharePoint a link will be provided to his or her My Profile page in their My Site. This means that if I see a document that interests me I can immediately click on the link to the contributor’s My Site to find out more about them. How good the information is on the profile depends upon whether the individual is motivated to contribute to it (though organisations can usually pre-populate basic role and contact data for staff from other databases).
• It helps internal communications: My Sites are places to which the organisation can push messages relevant to that individual. The individual can use their My Site to manage their subscriptions to other information sources in SharePoint, and to keep tabs on the various team sites that they belong to or are interested in.

A My Site is a SharePoint ‘site collection’ in its own right, which means that the individual has adminstration rights over it. They can add new pages, or new sub-sites. They can add whatever SharePoint web parts they wish to pages within their My Site. They can give trusted colleagues permissions to view and/or contribute to documents, document libraries, web parts, pages or sub-sites within their My Site.

The high degree of control that an individual has over their My Site has pros and cons.

• On the plus side it means that it gives the individual scope to experiment and customise and is a great place for them to learn about the functionality available in SharePoint, the way it works and what you can do with it.
• On the down side some organisations have found that underneath their My Site  individuals have created what are in effect team collaboration sites, visible only to the select group that the individual has invited in to that site, and evading any controls the organisations has put in place to govern and limit the creation of new team sites.  Organisations can mitigate this risk of this by imposing quota sizes on My Sites

Steve Gaitten has written an excellent critique of My Sites in SharePoint 2007, which is much fairer than its title ‘SharePoint My Sites Suck’ would suggest.

Gaitten’s comes up with three killer criticism of My Sites:

• Useability – Gaitten says My Sites are ‘too hard to personalise – if we’re asking users to choose, add and configure Web Parts to create a basic user profile, it’s just not going to work. Many employees can’t do it and most just don’t care enough to make the effort.’
• Lack of lifestreaming features – ‘Status is absent. The single most important feature driving the adoption of Twitter and Facebook is the status message. What are you doing right now?’
• ‘Lack of integration with existing social networks. Social networking is one of those things where you need a lot of people to play for it to work. On that basis alone, I’m skeptical that viable social networks can be formed outside of the largest organizations. You need thousands of users updating profiles, status messages and contributing to the network to keep it interesting. A company with even a few thousand employees may find that there just isn’t enough critical mass for an active social network to form’.

Gaitten gives the reason for the weakness of social computing within SharePoint as being the fact that the My Site feature must have been thought of and planned as early as 2005, when social computing was in its infancy. (SharePoint 2007 was released late 2006). This is a fundamental weakness of packaged software as opposed to applications hosted by the providor.  Think how many times Facebook has radically changed the look, feel and operation of its application since SharePoint 2007 was released. Even if SharePoint 2010 was to come up with a social computing facility that matched Linked in, Facebook, Twitter and iGoogle now, how will it keep up as those applications, and whatever succeeds them, continue developing?