What is SharePoint good for?

SharePoint is unique amongst information management systems in that it is rarely purchased with any specific purpose in mind. It is most often bought bundled in with other products when an IT department negotiates an enterprise agreement with Microsoft.

Microsoft enterprise agreement

This creates a challenge for such organisations. What should they use it for? What shouldn’t they use it for?

Giving people out of the box SharePoint team sites and hoping that they do something useful with them produces very variable results. Some teams will make their site work. Many others will either decline to put the effort in to tailor the site to their needs, or will tailor the site but make a bad job of it and the result will be unpopular with their colleagues. SharePoint team sites are most effective when they are targeted at (and restricted to) those areas of the business whose importance justifies the configuration of a template team site targeted specifically at their work.

The SharePoint Symposium in Washington DC that I attended last week brought together analysts such as Tony Byrne, Alan Pelz-Sharpe, Rop Koplowitz and Mark Gilbert; together with SharePoint implementers such as Shawn Shell and Richard Harbridge

SharePoint is a platform, not an application
Tony Byrne said SharePoint was best described as a platform rather than as an application. It had a great many features which provided organisations with the potential to build applications.

An application is a set of features that have been combined together to provide an organisation with a useful capability (a contracts management capability, a social computing capability etc.). SharePoint is feature-rich, but in most areas these features have not been knitted together in a way that provides an organisation with a useful capability.

For example in social computing SharePoint has every feature that you might ask for – blogs, wikis, microblogging, discussion boards etc.  The features may be variable in quality (Tony said that the microblogging ‘sucks’), but they are all there. However you could not roll out vanilla SharePoint blogs, wikis, discussion boards, activity streams etc. and expect any significant uptake of these features.  If you wanted a social computing capability that people would actually find useable, interesting and lively then you would either have to build a customised user interface on top of those features, extend SharePoint with a third party tool, or use a separate application entirely.

A member of the audience asked whether SharePoint could be used for business process management.  Tony said that SharePoint had good routing features, and if all the work in a particular process was to be done within SharePoint itself then it could fit the bill. But a business process management capability implies that a system is orchestrating a complex process across several different applications and SharePoint lacks the ability to do that.

The SharePoint ecosystem

Tony told us that consultants, ISVs and developers liked SharePoint because it ‘meant never having to say no to a client’. Given enough time and resource SharePoint could be made to do almost anything. SharePoint extends the .Net framework (Microsoft’s answer to Java, which is in itself a platform). It offers an object model and services on top of .Net that developers can make use of.  It is better documented than most other enterprise applications, and it’s codebase is remarkably stable considering the scale of the product. It also has an enormous ecosystem, and Microsoft themselves estimate that for every €1 spent on SharePoint licences €6 are spent with the SharePoint ecosystem.

Tony warned organisations against trying to use SharePoint to meet every technological need. Most organisations have SharePoint licences, and when a technology need arose someone, somewhere will ask the question ‘why not use SharePoint for that?’. Tony advised organisations to turn this question on its head and ask ‘why SharePoint?, ‘Why would customising and/or extending SharePoint be more effective than getting a tool that is already optimised to fulfil that purpose?’

Mark Gilbert said that organisations were getting frustrated at having to implement third party tools in order to do things that they expected SharePoint to be able to do natively.

Alan Pelz-Sharpe described the SharePoint ecosystem as being a very crowded market, full of small companies. There are lots of start-ups and many companies are operating on venture capital funding. Some of the companies will last but many won’t. Some will go-under, some will get acquired. When buying a third party product to extend SharePoint, organisations should consider the financial stability of the company and their road map for the future. Will they still be around to update their product every time a service pack is issued or a new version of SharePoint is released?

Using SharePoint to provide a user interface to data held elsewhere
Shawn Shell described some powerful uses of SharePoint’s Business Connectivity Services (BCS).  Organisations often have databases that contain valuable data (for example cusomer data in a customer database) that is not available to all staff because the organisation is reluctant to pay for extra user licences.  SharePoint’s BCS allows you to set up a connector to the database, and to surface the customer data as a SharePoint list.   This data can then be used to improve the findability of customer data in SharePoint. A column from the list such as ‘Customer ID’ could be used as a controlled vocabulary within the SharePoint environment, to be added as metadata to documents held in document libraries within the SharePoint implementation.

Office 365
Tony Byrne said that Microsoft seemed to have settled on a three year release cycle for SharePoint.  This three year wait was fine for steady areas like document management, but in quicker moving areas like social computing it put SharePoint at a disadvantage compared to more agile competitors.

Shawn Shell said SharePoint Online (a component of Microsoft ‘s cloud based Office 365) could give Microsoft the opportunity to introduce new features on a rolling basis rather than waiting for the next product release.   Shawn described SharePoint Online as a ‘weird mixture’ of the core SharePoint features (lists, libraries, sites etc.) available in SharePoint Foundation, together with some, but not all, of the features of the paid-for SharePoint 2007.

Rob Koplowitz said that early adopters are having a really hard time with SharePoint Online,  but he thought that over the long term it would be the way that most SharePoint clients will go.

Mark Gilbert said that at the moment Office 365 was geared to small and medium sized organisations.  At the Office 365 product launch Microsoft used the example of a dog grooming company.

SharePoint versus Box.net
Alan Pelz-Sharpe said that SharePoint is huge business and it is not going to go away any time soon, but cloud based file share services such as box.net, Huddle and others are the first to start to firing arrows across SharePoint’s bow. The reason why they are a threat is that filesharing is the core of SharePoint’s capabilities.  Filesharing is the one thing SharePoint does really well out of the box.

Alan said that  SharePoint Online would  find it hard to compete with born-in-the-cloud competitors.  Companies like Box.net  were cloud based from the start. They have optimised their product architecture for the cloud, all of their development effort goes into their cloud offering, and their partner channel is geared for the cloud.  In contrast SharePoint ‘s history is as an on-premise product.  Its architecture is geared to on-premise, the vendors in its ecosystem were mostly geared around on-premise.

Mark Gilbert said that SharePoint is an environment that organisations are used to tweaking and fine tuning, but you cannot do that to anything like the same extent with SharePoint Online.

Rob Koplowitz said that SharePoint was a Swiss army knife of a product that had a huge array of different features.  A service like Box.net was like a screwdriver – it did one job (filesharing).  But if you only want a screwdriver, why buy a Swiss army knife? The threat to SharePoint from Box.net and others would come if organisations decided that they wanted a way to tackle the fileshare problem without engaging with the complexity of SharePoint.

The analysts noted a trend in attempts to tackle the filesharing problem.  Early attempts to get away from shared drives came from powerful systems like Documentum that imposed rigid disciplines on users and gave strong central controls.  SharePoint upstaged these products by providing  a product that was simpler for end-users (at the expense of weaker central control and a more sprawling, less coherent repository).  Now Box.net comes along and offers a solution that is even more simple for end-users, and has even less central controls.

Rob Koplowitz pondered whether organisations were doomed to forever repeat the shared drive scenario on different products (SharePoint, Box.net etc.) with content sprawling on systems that are always beyond the organisation’s control.

Is SharePoint a records management system? – podcast

Last Friday Brad Teed (CTO of GimmalSoft) and I discussed whether or not SharePoint could be regarded as a records management system. We recorded the discussion for the ECM Talk podcast series.

Click on the play button below to hear the podcast. If the play button is not showing in your browser (it needs Flash) then you can get the podcast from here or from i-Tunes (search on ‘ECM Talk’). The podcast is around 45 minutes minutes long.

Brad said that SharePoint 2010 could be regarded as a records management system with the caveat that it did not do things in the way that traditional records management systems did them.

I conceded that SharePoint 2010 had records management features (such as holding and applying retention rules, holding a hierarchical classification, locking documents down as records) but I did not think that these features were brought together in a coherent enough way to justify calling SharePoint a records management ‘system’.

SharePoint 2010 offered organisations two different approaches to records management – the in-place approach and the records centre approach. Brad and I described and critiqued these two different approaches . I said it was a choice between ‘a rock and a hard place’ because both approaches had serious drawbacks:

  • The in-place approach left records scattered around team sites under the control of local site owners without providing any reporting capability to give a records manager visibility over them all
  • The records centre approach had the advantage of bringing records together into one place that the records manager could control. However it brought with it the complexity of managing the routing rules necessary to get documents from SharePoint team sites to the record centre

Brad and I will be debating the issue of records management in SharePoint live at the SharePoint Symposium in Washington DC on 2 November 2011.

Musing Over MoReq2010- podcast series with Jon Garde

There are many differences between MoReq2010 and previous electronic records management certification regimes (including DoD in the US, TNA in the UK, the previous version of MoReq in Europe etc.)

MoReq2010 is different in:

  • its fundamental assumptions (it assumes that records are captured in many different systems within an organisation, rather than in one records system)
  • what it is fundamentally trying to do (specify the minimum set of things that any application needs to do to manage its records, rather than specify a general records system that does everything any organisation would want its record system to do)
  • the concepts it uses  (most notably the concept of an ‘aggregation’ replacing the concept of a ‘file’ )
  • the way it is structured (with the functionality grouped into ten services)
  • the fact that it will develop over time (with new modules being added to meet to meet specific needs)

I spoke to Jon Garde (the lead author of MoReq 2010) before his talk to the 2011 IRMS conference.  He said that it was impossible to do justice to all those changes within the confines of a 30 presentation speech.  I suggested a podcast series in which we could discuss in detail the different areas of MoReq2010, and the thinking behind them. We have called the podcast series MusingOverMoReq2010.

We have recorded three epsiodes so far – episode 1 discusses the general philosophy behind MoReq 2010. Episode 2 looks at the classification service, and the role of a classfication within a MoReq2010 compliant system.   Episode 3 looks at the disposal service and how retention rules are applied within a MoReq2010 compliant system

As well as discussing an aspect of MoReq2010, each episode also contains news of recent developements around MoReq2010. This is to allow for the fact that MoReq2010 will continue to develop as extension modules are written, and as the testing centres get set up and certification gets underway. The third section of the show is a ‘postbag’ section where we will discuss any questions you send us.

You can listen to it on the Musing Over MoReq2010 website.   You can subscribe to it from the i-tunes store (search on Musing over MoReq2010) or by getting your podcatcher to subscribe to this feed: http://musingovermoreq2010.com/MusingOverMoReq2010_feed.atom

10 questions on the current state of the ECM market

On Wednesday I recorded an ECM talk podcast in which I put  10 questions  about the current state of the enterprise content management market to Alan Pelz-Sharpe.

Click on the play button below to hear the podcast. If you don’t see the play button (it needs Flash) then you can get the podcast from here. The podcast is just over 50 minutes long.

The ten questions, and a flavour of some of Alan’s answers are given below – there is a lot more detail in the actual podcast itself

Why have HP bought Autonomy?

Alan said that most analysts were surprised at how much HP paid for Autonomy.  The best guess at what HP (a hardware company) wants to do with Autonomy (a software company) is that they may wish to create some kind of appliance which has Autonomy’s IDOL search engine already loaded onto it (a bit like the Google search appliance).  One thing that HP and Autonomy have in common is that they have both bought well-regarded electronic records management vendors (Tower and Meridio respectively), and done very little with them.

How hard have the ECM vendors been hit by the rise of SharePoint?

Alan said that the ECM vendors haven’t bit hit as hard as you might think. Their revenues are still rising, and most of them enjoy good relations with Microsoft.

How does EMC and Open Text compare with the bigger ECM vendors (Oracle and IBM)

Alan said that Oracle and IBM are so big because they do a huge variety of stuff as well as ECM.  But at the end of the day if you are buying FileNet from IBM you are dealing with the FileNet division, not the whole massive company. So for buyers of ECM systems company size doesn’t matter that much.  Open Text is the largest company that focuses exclusively on ECM.   EMC’s business is mainly about storage.  They bought Documentum, but Documentum is very different from the rest of the EMC group and there has not been many synergies.

What is happening in the CRM (Customer relationship management) arena and how does it relate to ECM?

Essentially ECM and CRM are seperate worlds without much overlap.  CRM is a vital tool for many organisations.  As yet there is not a great deal of tie-ins with ECM.  Oracle has both a CRM and an ECM suite, which work together reasonably well.  SAP signed a large deal with Open Text but there doesn’t seem to be a huge number of organisations using SAP together with Open Text products.  Many of the CRM tools will do a little bit of document management of customer related documents, but for the most part organisations will have CRMs that don’t talk to whatever ECM product(s) they have

The Europeans have just revised their electronic records management specification (MoReq2010).  When will the US records management standard DoD 5015 be revised (it was issued back in 2007)

Alan said he didn’t know of any plans to revise DoD 5015.  SharePoint drove a horse and cart through DoD 5015 because Microsoft made the decision to release a document management product that did not comply with it but had huge market success.  Alan said he didn’t see the point in revising it –  it was specifically tailored to US Government (DoD stands for Department of Defense) so some of the requirements are overkill for organisations in other sectors.

After the podcast it occured to me that there was no need for DoD 5015 to be revised.  MoReq2010 is the first of the electronic records management specifications to be extensible. Rather than revise DoD5015, if there were requirements specific to the US  (or to particular sectors in the US) that were not covered in the core requirements of MoReq2010 then a seperate module could be written to cover those requirements for vendors wishing to target their products to the US market.

What is happening in the intranet arena?

Alan said that nothing dramatic is happening in the intranet arena.  Some intranet makeover projects will have been hit by the economic downturn.  Alan can’t understand why some organisations want to use the same product to manage there external web-site and their intranet – to him they are fundamentally different things with different requirements.

Do you know any organisation that manages their e-mail well?

Alan said that of all the ECM implementations that he sees, the type that gives the quickest and most reliable return on investment is an e-mail archiving tool brought in to take stored e-mails off the mail servers.  I said I would like to see some of the e-mail archiving vendors apply for certification for their products under MoReq2010, so that buyers could be more confident of their ability to export e-mails out of their e-mail archive if they neeeded to.

What do you think of PAS 89?

PAS 89 will be a UK standard on enterprise content management, with a view to becoming an international standard.  Alan said PAS 89 was a good attempt to define the scope of enterprise content management, although it was hard to think of what an organisation would specifically use it for.

How does Alfresco compare with the proprietary ECM products 

Alan said that if we were talking about open source ECM products Nuxeo should be mentioned alongside Alfresco. Both of them are established, mainstream enterprise content management systems.  The main difference between them and the proprietary ECM products is the licensing model.

How does Google Apps compare with the established ECM products 

In terms of impact on the ECM market Alan is more interested in Box.Net than Google Apps.  Alan and James discussed the prospect of new start ups deciding not to set up shared drives and instead using services like Box.Net in the cloud to provide a relatively simple place for colleagues to store and share documents.

After the podcast it occured to me that for a good 15 years we have been wondering what would replace shared drives. Shared drives have survived so long because anything that could have replaced them for general document storage (EDRMS, SharePoint) has proved more complex than shared drives, and so shared drives retained their role as an uncomplicated, quick place to store documents.  From a users perspective something like Box.net is as simple to use as a shared drive, and has the advantage that folders and documents can be shared with people outside the team, and with people outside the organisation.   From an organisation’s information management point of view box.net is currently little better than a shared drive in terms of being able to apply retention rules and a records classification (though maybe if an ecosystem grows around Box.net someone could come up with a MoReq2010 compliant plug in for it- that would be interesting!)


The implications of MoReq2010 for records management practice

The DLM forum is having its triennial conference in Brussels this coming December.  I responded to the call for speakers with the following submission:

MoReq2010 can be seen as an attempt to ensure that the records management baby is not chucked out with the EDRMS bathwater.

The EDRM idea was solidly based in records management theory, but lost its market viability after 2008 thanks to the global economic downturn, the rise of SharePoint, and perceived problems with usability and user acceptability of EDRM systems and their attendant corporate fileplans.

SharePoint 2007 and 2010 both offer records management features, but neither offers a well-thought through records management model.  Most organisations with SharePoint implementations have not attempted to use records management features such as the SharePoint records centre.  Of those that are trying to use those features only a relatively small number will be able to impose sufficient governance to enable them to viably manage records in SharePoint.

For most organisations SharePoint will be a records management problem rather than a records management solution.  In a few years time more organisations will be saying ‘we need help managing the records that are scattered around our SharePoint implementation’ than will be saying ‘thanks to SharePoint content types we can now apply retention rules to records across our organisation’.

MoReq2010 doesn’t kill off the EDRM model (you can  use a MoReq2010 compliant system as an EDRM provided it complies with the plug-in modules for a user interface and a hierarchical classification), but it does not attempt to revive it either.

The fact that MoReq2010 is offering two alternative to EDRM, rather than just one, whilst continuing to support the EDRM model itself, indicates that the profession is not yet ready to commit its weight behind one single approach.  It also means that we are in a transition period, during which many records managers and consultants will be uncertain as to what approach to advise their organisations to take.

The two new approaches offered by MoReq2010 are ways of dealing with the ‘multiple repository problem’ – the fact that every organisation deploys numerous different applications to create, capture and store content and records.   EDRM systems rarely tackled that problem.  They typically relied on colleagues voluntarily declaring material into the EDRM as records, and there was rarely any incentive for colleagues to move documents out of a line of business application into an EDRM.

The back-end repository approach

The first of the two approaches is what I would call the back-end repository approach (I would like to call it repository-as-a-service but I fear you may mistake it for a cloud offering).  In this approach a MoReq2010 compliant system governs content captured in the multiple different applications of the organisation.  It governs either by taking that content out of those applications and storing it in the MoReq2010 compliant system, or by protecting and governing that content whilst it stays within those applications themselves.

This is an approach that vendors have been working on over the past five years – both EDRM/ECM vendors looking for ways to continue selling to customers who have chosen SharePoint instead of an EDRM, and e-mail archiving vendors looking to expand the scope of their archiving systems.  It is also compatible with service orientated architecture of IT departments, but no-one knows yet how it will play with moves to the cloud. MoReq2010 for the first time offers a certification regime for vendors taking this approach, giving the approach more gravitas and credibility, and offering buyers reassurance that their back end repository/archive will not in itself become a black hole from which it is hard to migrate records.

The back-end repository approach significantly changes the role of the records manager.  In EDRM implementations the records manager was interacting with users, training them, cajoling them, tackling change management challenges, and designing classifications that end-users would directly interact with   In the  back-end repository model the records manager has a different role – connecting legacy applications to the back end repository, and trying to ensure that no new application is deployed into the organisation unless it hooks into the back-end repository from day one.  The interaction with, and impact on, end-users will inevitably be reduced, but it is to be hoped it won’t be eliminated entirely.   It will still be important for end-users to be aware of whether or not a piece of content that they have contributed to a particular application has been captured by the back-end repository.

The in-application approach

The second of the two approaches is the addition of records management functionality to each  application deployed in the organisation so that these applications can manage their own records.

This is the approach that I sense the authors of MoReq2010 would like to see prevail in the world. They are well aware that every time a record moves from system to system it loses context, and that ideally records management metadata would be captured from the moment a record was first captured into an application.

This approach is beyond the capabilities of any single organisation – no organisation could customise all their applications for them to become MoReq2010 compliant.  It becomes viable only when the vendors of line of business systems, make their products MoReq2010 compliant – whether they be sector specific applications like social care systems for local authorities, or line of business applications like HR systems.  Its a battle worth taking on for the profession, and worth fighting, but success is likely to be patchy.  The hope is that a tipping point could be reached when everybody expected every application to be MoReq2010 compliant, and felt that something is wrong if it was not compliant.

Preserving e-mail – records management perspectives

On July 29 I attended the Digital Preservation Coalition event Preserving e-mail – directions and perspectives . The event brought together records managers, archivists, cultural heritage institutions and digital preservation experts.  For a summary of the event see Chris Prom’s Practical e-records blog (starting here).  In this post I will give some thoughts on the records management perspective at the event

Three approaches to managing e-mail

In the afternoon tea break Stephen Howard   gave me his take on the three different approaches records managers could take to e-mail

  • the message-by-message approach – where users are encouraged to move significant e-mails out of their e-mail client and put them together with other documents arising from the same work (this is the traditional records management approach).
  • the e-mail account by e-mail account approach – where some individuals within the organisation are selected as having particularly important roles, and their entire e-mail account is preserved
  • the whole e-mail system approach – where the organisation treats its entire e-mail system as one aggregation and applies one retention or preservation rule to the entire system

In his current organisation Stephen is thinking of applying the in-box by in-box approach. It would be relatively easy to identify people in key roles whose e-mail was worth preserving. Those individuals could be told of the organisation’s intention to preserve the contents of their e-mail account after they had left. They could be given ways of filtering out personal e-mail so that the personal stuff did not enter the archive.

Earlier in the day Stephen had given a presentation in which he reflected candidly on advice he had given back in 2005 to a local authority he worked for at the time. The head of IT in the authority was concerned about the e-mail servers, and their lack of resilience in the face of mounting volumes of traffic and e-mail storage. They wanted to buy an e-mail archiving tool, to remove stored e-mails from the production e-mail servers.  Stephen at the time advised them not to.

The authority decided against an e-mail archive.  Instead they adopted the intention of implementing an electronic document and management system (EDRMS) to manage records in all formats, including e-mail.   In the meantime they  used an array of methods to encourage colleagues to adopt better e-mail practice.  The authority :

  • asked colleagues to save significant e-mails into shared drive folders
  • put quotas on e-mail in-box sizes to encourage staff to weed out ephemeral e-mails
  • encouraged people to avoid sending attachments where alternatives existed
  • gave advice and training on good use of e-mail

None of these measures did any harm, but the overall approach did not work. Few colleagues saved e-mails into the shared drives. The bottom fell out of the EDRM market and the EDRM never came. Stephen wondered whether the IT manager was right after all – maybe the e-mail archiving tool would have been the least-worse option.

Records management concerns about e-mail archiving tools

Records managers have had philosophical concerns about e-mail archiving tools.

A standard definition of a record is that it consists of all documentation regardless of format needed as evidence of a piece of work.  The idea of treating one set of documentation (e-mail) differently purely because of its format was anathema to us records managers.

There are practical as well as philosophical concerns.  In particular the concern that an e-mail archive operates as a ‘black hole’ .   Such an archive may well have a great search engine, but how could the organisation allow people to use that search engine given the vast amounts of personal information buried in every e-mail account?  The fundamental problem is that a typical e-mail account  makes no differentiation between innocuous e-mails, and e-mails containing sensitive personal information about the e-mail account holder or the people they correspond with.

In practice an organisation could allow:

  • individuals to access e-mails in the archive that were sent to or received by themselves
  • central administrators to search the entire archive for e-mails that fall within the scope of a legitimate e-discovery request, data protection subject access request or Freedom of Information request

But I don’t see how an organisation could allow staff to search across an e-mail archive  on a day to day basis, to answer mundane business questions, because it would then also be possible for them to  search for personal information on particular colleagues.

Yes you could tell staff that if they send or receive e-mails containing sensitive personal information about themselves or third parties then they should delete it from the e-mail archive, or flag it up with an access restriction.   But could you ever be confident enough that this has been acted upon to widen up access to the e-mail archive?

The access permission problem means that  organisations will not want to give up totally on the idea of having e-mails aggregated in some ways, other than simply lumped together in an e-mail archive divided into individual e-mail accounts.  One of the aims of Customer Relations Management (CRM) implementations  is to ensure that e-mails to/from customers are aggregated by customer rather than by the e-mail accounts of the members of staff that sent/received them.  An EDRMS implementation aims at aggregating e-mails and other documentation according to the piece of work that they arose from.   Both approaches offer the advantage that access permissions can be ascribed that fit the nature of those e-mails.

We need to have our cake and eat it.  The advantage of e-mail archiving tools is that they give you the security of  knowing that you have a record of everything that has come in or out.  But what we also need is the ability to apply frameworks that enable those e-mails to be understood, managed and accessed according to different criteria than the name of the individual who sent or received them.

What impact will MoReq 2010 have on the e-mail archiving tool market?

Until recently the records management world treated the existence of multiple applications within an organisation (e-mail clients, line of business applications, CRM systems, an HR system etc.)  as a problem that could best be mitigated by implementing a single electronic records management system and endeavouring to get all documents or e-mails needed as records saved into it.

The recent MoReq2010 specification takes a different approach. It attempts to boil down a core set of records management requirements with a view to making it feasible for any and every business application to have enough records management functionality to manage its own records, and to export those records and accompanying metadata, rules and classifications at the end of the useful life of the application.   This is hugely ambitious, as line of business application developers and vendors rarely take notice of records management specifications.

The first output of  MoReq2010 – the Core services and Plug-in Modules published in June of this year, does not specifically mention e-mail.   This is because the Core services cover only that minimum set of requirements that every records system  should possess, and it is possible to envisage a records systems that is not intended to hold e-mails.  But an extension module of MoReq2010 is planned, to specifically outline MoReq 2010 records management requirements for e-mail.

It will be interesting to see what effect that MoReq2010 e-mail module, when it appears, will have  on the e-mail archiving tool market.  A MoReq 2010 compliant e-mail archiving system would be an interesting proposition for records managers – I wonder if any of the big players in the market will rise to the challenge.

This 2009 Gartner magic quadrant report on e-mail active archiving tools shows that many such tools are branching out from simply archiving e-mails and now claim to be able to archive and manage material in shared drives and in SharePoint sites.  All the more reason for such products to go for MoReq 2010 certification.

Whether they do go for it depends in part upon their willingness to re-architect the way their systems maintain metadata and event histories.  MoReq 2010 is much more prescriptive on these fronts than previous standards, and established players with set architectures may be reluctant to change.

The challenges of archiving databases – Podcast with Kevin Ashley

There is seen to be a big divide between the domain of ‘structured data’ sitting in rows and columns within databases, and that of ‘unstructured data’ (documents, images etc). In fact the two domains are related.  Every document management system has a database within it to maintain the metadata about the objects it holds.  It is worthless to be able to migrate/extract/preserve the documents unless you can also migrate/extract/preserve the metadata held in the database.

Last Friday I had a discussion with Kevin Ashley  about digital preservation and the challenges of archiving data from databases.

Archivists encountered the problem of archiving databases earlier than they faced the problem of archiving electronic documents from document management systems.  This was simply because organisations used computing power for structured data  earlier than they did for the creation and storage of documents.  In the early 1980s Kevin was already involved in attempting to rescue data from legacy databases within a UK government research council.

In 1997 Kevin led the creation of NDAD, the National Digital Archive of Datasets.  This pioneering service, was set up under contract to the UK National Archives (then called the Public Record Office), and opened in 1998.

At the time it was widely thought within organisations that databases were the sphere of IT professionals and data managers, not records professionals (records managers/archivists). In fact NDAD needed the skills of all three of these professions. Kevin said that the contribution that the archivists in NDAD made was invaluable, because they knew how to to draw up agreements with the contributing organisation, and how to capture the context of the dataset (who created it, why, how, what they used it for etc.).

The National Archives required NDAD to use the ISAD G standard of archival description to catalogue each dataset. I asked Kevin whether ISAD G had been difficult to adapt to structured datasets (it was written with files/documents in mind). Kevin said that ISAD G had been very useful and worked very well for the datasets.

I asked Kevin whether he thought that the databases in use by organisations today were easier or harder to archive than the databases in use in the 1990s.   Kevin said that the challenges were different.  An individual database in an organisation today was easier to understand and extract data from than an equivalent database in the 1990s. But the challenge today is that the databases in an organisation tend to be integrated with each other. For example all or most databases in an organisation may use the organisation’s people directory to hold information about their users. As soon as you try to archive data from one database you are faced with the challenge of archiving data from all the other databases that it drew data from.

I asked Kevin whether initiatives such as the open data initiative at data.gov.uk and the whole linked data/semantic web movement would mean there is less of a role for archivists in getting government datasets into the public domain. He said that was very rare for an organisation to make an entire database available to the public online. Usually the public would be given access to a derived database, which contains only a subset of the data in the database used by the government department itself. So there is still a role for the archivist in ensuring that the database that actually informed government decisions was preserved.

Kevin talked about one of the perennial challenges of digital preservation, the challenge of avoiding lock-in to an application and ensuring that information (whether it be structured data in a database, e-mails in an e-mail system, or documents in a document management system) can be exported from a particular application when that application is no longer used by the organisation.   I was interested in the parallels here with the recently published MoReq 2010 records management standard.

MoReq 2010 was written on the premise that the content of applications is typically still of value after the application itself has fallen into disuse, and therefore a key attribute of any application must be that it can export data, objects and metadata in a form that another application can understand.  After we had stopped recording we realised that link between NDAD and MoReq 2010 comes in the person of Richard Blake, who has recently retired from the the UK National Archives. Richard was strongly involved in both the creation of NDAD in the 1990s and with the writing this year (with Jon Garde, Richard Jeffrey-Cook and others) of MoReq 2010.

Kevin reported Richard as saying that one of the weaknesses of early electronic records management system specifications (such as TNA 2002 from the National Archives) was that although all compliant EDRM system would keep metadata in a way that could be exported, each different EDRMS kept metadata in a different way, so it was hard an organisation to migrate from one vendor’s EDRM system to another. It was this experience that informed MoReq 2010s decision to define very precisely how systems keep metadata, in order that the metadata of one MoReq 2010 compliant system could be understood by any other MoReq 2010 compliant system.

Click on the play button below to hear the podcast. If you don’t see the play button (it needs Flash) then you can get the podcast from here. The podcast is 44 minutes long.

Notes

This podcast was recorded for the Records Management Today podcast series, hosted by the Northumbria University. To see all the podcasts in the series visit Records Management Today

Elizabeth Shepherd and Charlotte Smith have written an article on the application of ISAD G to archival datasets, which focused on NDAD’s experience. (Journal of the Society of Archivists ‘The application of ISAD(G) to the description of archival datasets’ Journal of the Society of Archivists 21: 1 (April 2000): 55-86)

Patricia Sleeaman, Digital archivist at the University of London wrote an article for the journal Archivaria about the National Digital Archive of Datasets, the article is available as a free pdf download from here

In the podcast Kevin discussed two datasets accessioned by NDAD:

Kevin Ashley is Director of the Digital Curation Centre.  He is @kevingashley on twitter

Update on MoReq 2010 from the DLM Forum meeting in Budapest

I attended the DLM Forum meeting in Budapest last week (May 12 and 13) at which Jon Garde announced that the core requirements of the MoReq 2010 specification had been finalised and would be published as a PDF on the DLM forum website within the fortnight following the meeting. It was possible that it might also be issued as a hard copy publication later in the year.

How the final version of the MoReq 2010 core requirements differs from the consultation version issued late last year

Jon Garde described the changes to the core requirements of MoReq 2010 since the
consultation version was released late in 2010. These changes included the adoption of a service orientated architecture for MoReq 2010, the dropping of the notion of a primary classification, and a reduction in the number of requirements.

Adoption of a service orientated architecture model
All the requirement in the MoReq 2010 core requirements have been bundled into ten services. A MoReq 2010 compliant system will be capable of offering up its functionality as services, that could be consumed by one or more other information systems within the organisation.

For example several records systems within an organisation could all consume the classification service of one MoReq 2010 compliant system, enabling the organisation to hold its fileplan in one place whilst having it used by several systems.

A MoReq 2010 compliant system must possess the capability to provide ten services:

  • a records service (the capability to hold aggregations of records)
  • a metadata service (the capability to maintain metadata about objects within the system)
  • a classification service (the capability to hold a classification, to apply it to aggregations of records, and to link headings within the classification to retention rules)
  • a disposal service (the capability to hold retention rules, and to dispose of records in accordance with retention rules)
  • a disposal hold service (the capability to prevent the application of a retention rule to a record, for example because the record is required in a legal case)
  • a search and report service (the capability to retrieve and present records and metadata in response to queries)
  • a user and groups service (the ability to maintain information about people and groups that have permissions to use the system)
  • a role service (the ability to assign roles to people and groups to determine what those people and groups can and can’t do within the system)
  • system services (the capability to maintain event histories in relation to objects held within the system)
  • an export service (the capability to export records together with their metadata and event histories in a form that another MoReq 2010 compliant system could understand)

Abandoment of the notion of a ‘primary classification’

The notion of a ‘primary classification’ for records (see my previous post) had been dropped. Instead a record will be assigned a classification, from which it would by default inherit a retention rule. It would be possible though for a person with appropriate permissions to override that inherited retention rule, and instead assign to the record a different retention rule, or to get the record to receive a retention rule from a different part of the classification scheme to the one it has been assigned to.

Reduction in the number of requirements
The number of requirements had been significantly reduced. The consultation draft had contained 436 requirements, these have now been consoldated into 170 requirements. But the final core requirements document would be longer than the consultation draft, because the introductory explanations had been increased to 90 pages.

Plans for the future development of MoReq 2010

The MoReq Governance Board has ambitious plans for the development of MoReq 2010, and regarded the publication of the core requirements as only the beginning. MoReq 2010 has a modular structure, and additional modules are planned that vendors may choose to submit their products for testing against.

The  DLM forum are planning to have a first wave of additional modules for MoReq 2010 available by the time of their triennial conference (due to be held in Brussels in the week of December 12, exact dates/venues yet to be announced).  Unlike the core requirements, the additional modules will be optional rather than mandatory.

Included in the first wave will be:

  • an import service – providing the ability to import records and associated metadata from another MoReq 2010 compliant system. Note that the ability to export records is a core requirement, but the ability to import records is an additional module.  This is because an organisation implementing its first MoReq 2010 compliant system does not need that system to be able to import from another MoReq 2010 compliant system.
  • modules that provide backwards compatibility with MoReq 2

Backwards compatibility with MoReq 2 is important.  One European country (the Czech Republic) has enshrined compatibility with MoReq 2 into records management legislation. The modules that will give backwards compatibility to MoReq 2 will be:

  • a scanning module
  • a file module (MoReq 2010 replaced the concept of the ‘file’ with the broader concept of an ‘aggregation’. The additional module would ensure that a system could enforce MoReq 2 style ‘files’ (which can only be split into volumes and parts). In MoReq 2010 terms a MoReq 2 file is simply one possible means of aggregating records
  • a vital records module
  • an e-mail module (the core requirements of MoReq 2010 itself talks generically about ‘records’ and do not focus specifically on any one particular format)

Note that a system could be MoReq 2010 compliant without being MoReq 2 compliant (because the additional modules that give MoReq 2 compliance are voluntary and not part of the core requirements of MoReq 2010). Any organisation that wanted MoReq 2 compliance as well as MoReq 2010 compliance would be able to specify that a product must be certified against those additional modules.

It is hoped that more additional modules would follow. Jon would like to see MoReq 2010 additional modules that cover records keeping requirements in respect of cloud computing, mobile devices and social software. He urged anyone who feels that there are needs that MoReq 2010 could usefully address to come forward and develop a module to address those needs. For example modules that provide functionality specific to a single sector (health sector, defence sector etc.).

There is also the possibility that modules could be written to specify the functionality required for a MoReq 2010 compliant system to also demonstrate compliance with a different standard or statement of requirements. For example a module could be written to ensure that a MoReq compliant system met all the requirements of the US DoD 5015.2 specification (which raises the interesting possibility of a European testing centre announcing that a system is compliant with the US records management specification).

Development of test centres

;

The MoReq Governance Board plans to accredit an international network of testing centres, to whom vendors can submit products for testing against MoReq 2010. Six organisations have already expressed an interest in becoming testing centres. There is no limit to the number of test centres that may be established. The test centres will use test scripts and templates created by the MoReq Governance Board. Vendors will pay a fee to the test centres to have their products tested, and (assuming they are successful) a fee to the DLM Forum to validate the recommendation of the test centre and to award the certificate.

As well as vendors submitting their products for testing, it would also be possible for an organisation to submit their specific installation of a system for testing.

How MoReq 2010 differs from previous electronic records management (ERM) system specifications

On 5 April this year I sat on a panel at the UK Information and Records Management Society (IRMS) conference. We were each asked for our view of the future of technology. We talked about the cloud, and also of the end of the dominance of ‘the document’ as a format. One member of the audience brought us back down to earth with the comment ‘never mind the future, we don’t know how to manage records in all the many and various systems our organisations have now’.

Ed Fowler of Cap Gemini said that organisations will need to get used to the fact that their records will be created and stored in many different content repositories, and that few if any of these repositories will have records management functionality.  Ed said records managers should abandon any hope that they could specify and implement one corporate electronic document and records management system (EDRMS) and expect all their colleagues to save their records into it. Recently he had only seen one UK organisation tendering for an EDRMS.

Ed was announcing the death of the traditional EDRMS. I would add a caveat here – information systems don’t die, they just lose momentum. EDRMS has lost momentum but most of those organisations that have succeeded in implementing them corporate-wide have felt a benefit and will continue to maintain and develop their systems.

The loss of momentum in EDRM is nowhere better illustrated than with MoReq, the European Union’s specification of electronic records management (ERM) systems. In early summer 2008 MoReq 2 was launched, only to be scuppered by two tornadoes: the global economic downturn, and the rise of Microsoft’s SharePoint (which took the collaboration space out of the reach of EDRMS vendors). By the end of the following year, 2009, only one vendor had submitted a system for testing against the specification, and the decision was taken by its sponsors, the DLM Forum, to commission a radical rewrite to the MoReq specification. The rewritten specification, called MoReq 2010, will be launched at the next meeting of the DLM Forum, in Budapest, next week.

The previous day (April 4) of the IRMS conference had seen a debate between Jon Garde (author of the new MoReq 2010), and Marc Fresko (author of its predecessor, MoReq 2).

Jon Garde introduced MoReq 2010 by arguing that we needed to revisualise the ERM system.  Previous specifications saw an ERM systems as a stand alone content repository that stood alongside the other content repositories within the organisation. MoReq 2010 sees ERM as a capability that could be embodied within each separate application that the organisation uses, or that could sit behind those applications and manage records created within them.

Marc Fresko was critical of MoReq 2010. In his view it introduced too many new concepts too quickly and the timeframe for the publication of the specification has left too little time for debate.

The fact that Jon and Marc differ so sharply in their viewpoints shows how different MoReq 2010 is from previous versions of MoReq.

Previous versions of MoReq aimed to specify systems that could serve all the records management needs of any organisation, in any sector. They created the breed of information system called the EDRMS. The idea was that an EDRM system would be rolled out to all users, and would hold within it a hierarchical classification (called a business classification or fileplan) that covered all the work of the organisation. Users would capture and store documents and e-mails needed as records into the EDRMS, within files (folders) classified against the business classification/fileplan.

The traditional EDRMS could not on its own solve the problem of multiple repositories. In organisations some functions (types of work) are more important than others. The more important functions will usually have a line of business application procured or developed specifically for them. An EDRMS, because it aims to cover all the diverse activities of an organisation, by definition cannot be tailored to any one type of work.  This means even those organisations with corporate-wide EDRM systems still get the problem of managing records in multiple repositories. Once SharePoint came along this problem was made even worse – if an organisation is putting a lot of time and effort implementing SharePoint for collaboration they will probably not have the capacity to implement a corporate EDRMS in tandem with it, however much they recognise the problems of SharePoint sprawl and the weaknesses of SharePoint’s own records management model.

MoReq 2010 has been written to encourage different models of records management system to emerge. It does this by adopting a modular structure. All MoReq 2010 compliant systems have to comply with a core set of requirements. Beyond that there are different modules which vendors can chose whether or not to submit their product for testing against. Some vendors may well decide to submit traditional EDRM-style systems for certification. But they are also able to submit systems for testing that meet a completely different model, for example:

  • Systems that end-users do not interact with directly, but instead capture and store records that users had created in other systems
  • Systems that do not store records, but instead govern and protect records held in other systems
  • Line of business systems or single purpose applications that are not intended to be a general records system but which have have the ability to manage the records that they capture
  • Systems that can fulfill two or more of those roles – for example a system that could be deployed as a traditional EDRM that some end-users would interact with directly, but which also possessed the capability to manage records held in other content repositories

The old aspirations that MoReq 2010 abandons

MoReq 2010 is a major change from all previous specifications of electronic records management systems. It is a break not just with the previous two versions of MoReq issued by the European Union (MoReq and MoReq 2) but also with the UK’s TNA 2002 standard and the US DoD 5015.2 standard.

MoReq 2010 abandons three big aspirations that all those previous specifications shared. It abandons:

  • the idea of the ‘file’ in the electronic records management system being the ERM system equivalent of the traditional hard copy file MoReq 2010 replaces the concept of the ‘file’ with the new concept of an ‘aggregation’ This is a major break with the vocabulary of the hard copy era. In a MoReq 2 compliant system a ‘file’ was limited to two levels of hierarchy beneath it (file/sub-file/parts). In a Moreq 2010 compliant system an aggregation can have any number of levels of hierarchy. The MoReq 2010 ‘aggregation’ has a different relationship to a business classification/fileplan than a MoReq 2 ‘file’. The MoReq 2 file sat at the very bottom of the business classification/fileplan. An aggregation can be a multilevel hierarchy in its own right, so it can sit separate from the business classification, whose role is not so much to act as the only means of navigating around the records system, but more to apply retention rules to records.
  • the idea of one monolithic corporate business classification/fileplan universally applying retention and access rules to all records In previous versions of MoReq only one classification could be linked to retention rules and used to apply retention rules to records (a hierarchical corporate business classification, also called a fileplan). In Moreq 2010 compliant systems there is the possibility of having several classifications, any number of which can be used to apply retention rules to records. If a record is classified against more than one classification then one of them should be nominated as the ‘primary classification’ for that record. The primary classification is the one that the record inherits its retention rule from.
  • the idea of their being a single type of ERM system that can meet the needs of all organisations in all sectors Jon Garde identified that MoReq 2 was twice as long as the first MoReq. Jon says the reason for this is that both MoReq and MoReq 2 tried to specify systems that could meet the records management needs of all organisations, whatever their sector. This meant that if any sector of the economy could prove they had a genuine records keeping requirement then MoReq would try to incorporate it into the specification. The disadvantage of this is that vendors had to configure their systems to cover all sectors and all eventualities, which pushed up the cost for them of developing compliant systems. MoReq 2010 has been written so that the core module contains only requirements that are common to all or most organisations. If a sector has specific requirements they are able to write a separate MoReq 2010 module to capture those needs. Vendors that wished to target that sector could add that functionality to their system and ask for it to be tested and certified against that module. Vendors that didn’t wish to target that sector could ignore it. Jon Garde argues that the MoReq 2010 will be more sustainable over time than previous specifications, as new needs can be incorporated into new modules without having to republish the whole specification. Marc Fresko gave the counter argument that the structure of MoReq 2010 will be more complex for records managers in organisations to work with, because they are going to have to decide which modules are important enough for their organisation to insist upon.

The new aspirations of MoReq 2010

In place of those abandoned aspirations MoReq 2010 brings in three new aspirations:

  • the idea that the electronic records management system will sit behind the existing applications used to create and capture records in the organisation A system can be MoReq 2010 compliant without having any user interface. The core module merely insists that if a system does not have a user interface it must have an API (application programming interface) to enable it to integrate with those systems in the organisation that users do interact with to create records. Note that integrating an electronic records management system behind existing content repositories poses very different challenges to those posed by implementing a stand alone EDRMS that end-users directly engage with to save their records. It is likely that your various content repositories will have been procured from various vendors, written in various programming languages, and have varying quality APIs. Each integration to each content repository is a project in its own right. Aside from the technical challenge there is also the semantic challenge – each of those content repositories will have their own way of organising and describing content.
  • the idea of MoReq compliant systems controlling records held in other systems a system can be MoReq 2010 compliant without having the ability to store records, provided it has the ability to apply classifications, access rules, and retention rules to records stored in other systems; provided it can protect those records from deletion or amendment; and provided it can maintain (and export) an audit trail of events that happen to those records.
  • the idea that when an organisation changes its records system the new records system could understand everything that had happened to those records in the outgoing records system. Record systems capture records and apply retention rules to them. Jon Garde points out that the span of years of many of these retention periods is longer than the life of most records systems. Jon wants a MoReq 2010 compliant system to be able to export a record and its associated metadata at any point during the retention period, so that the receiving system could understand what retention policy has been applied to the record, when that retention period started, and how long it had left to run. The new records system could then apply the remainder of the retention period. MoReq 2010 is much stricter than previous specifications on how a system maintains its audit logs. The system must be able to export an event history for every object in the system.  The event history must record everything that has happened to that object since it was created in or captured by the system. In order for this event history to be understandable by another system MoReq 2010 insists upon the use of unique identifiers. A MoReq compliant system needs to allocate unique identification numbers to every type of object (records, aggregations, retention policies, classification headings etc.); to every actual object; to every type of event in the system (for example ‘the title of a document is changed), and to every actual event.
  • the idea of line of business system vendors incorporating records management functionality into their systems By stripping down the core requirements to a very minimum the DLM-forum hope that providers of line of business systems will see that it is possible for them to seek MoReq 2010 certification.  The hope/dream is that providers of finance systems, human resource management systems, facilities management systems, and customer relationship management systems will add MoReq 2010 functionality to their applications . Note that vendors of those kinds of applications won’t start to think about MoReq 2010 compliance unless and until purchasing organisations start specifying a preference for MoReq 2010 compatible applications. The challenge here is that records managers often do not have a voice in those kind of procurement exercises.

The likely impact of MoReq 2010

MoReq 2010 will bring interesting effects for records managers. The standards we were used to in the first decade of the 21st century (TNA 2002, DoD 50.15, MoReq 2) all ended up certifying a very similar type of system, implemented in a very similar way. Not quite ‘seen one EDRMS seen them all’ but almost.

Out of MoReq 2010 will sprout a variety of different applications, which will need a variety of different implementation methodologies. This is a challenge for the wider records community, not the sponsors of MoReq 2010. All MoReq 2010 can do is to encourage vendors to provide applications that manage records created and/or held in different content repositories (and provide certification to verify that they have that capability).

Whether or not MoReq 2010 ends up being successful will ultimately depend on whether we records managers can come up with viable implementation approaches for the type of application that it will certify.

Differences between SharePoint as an intranet platform and SharePoint as a collaboration platform

I spoke about SharePoint to an audience of intranet managers yesterday. The talk was at J Boye’s ‘SharePoint at work’ event in Copenhagen.

I made the following points:
• SharePoint is not the first platform that an organisation could use both for their intranet and for their general collaboration/document management environment. But it is the first platform to be market leader in both spheres.

• Managing a collaboration/document management environment is a different ball game from managing an intranet environment.

• The scale is different. In an intranet environment you can expect a sub-section of the organisation to contribute a small amount of content on a monthly basis. In a collaboration/document management environment you may reach the point where most of the organisation are contributing on a daily basis.

• The risks are different. In the collaboration/document management environment there will be material with sensitivites (personal/commercial/political). Such material is not typically contributed to intranets.

• The difference between the functionality of SharePoint when used for an intranet and the functionality of SharePoint when used for collaboration is paper thin. In either case you store content within SharePoint site collections. If you intend a site collection to be used for the intranet then you use the publishing site template to create the top level site of the site collection. If you intend a site collection to be used for collaboration you create the top level site with a different template (for example a team site template).

• Organisations typically put site collections intended for the intranet in a separate web application to those used for collaboration, with a separate URL address, different branding and different administrators and administration settings.

• There is no licensing difference between SharePoint Server 2010 used for an intranet and SharePoint Server 2010 used for collaboration/document management. Where an IT department have purchased SharePoint Server 2010 licences (for example as part of an enterprise agreement with Microsoft) it is hard for either intranet managers or records managers to explain why they want to buy and use a different system

• There may well be synergies to be derived from having both intranet and document management happen in the same environment, but realising these synergies is not easy. An organisation might identify valuable type of information (for example research reports), define a content type for it and then surface such material from the collaboration environment onto the intranet.

• If an organisation wants to use content types to leverage and make the most of valuable types of information then it becomes very important to control their content types. There is a tendency for content types, like everything else in SharePoint, to sprawl. Content types can be created in any site by any site owner, and be available not just in that site but in all sites underneath it. The more content types you have the less chance there is of a team choosing appropriate content type(s) for their document libraries.

• Your records manager may want to use content types to apply retention rules and/or to route documents to a records centre. If so they will want to apply content types to large aggregations of documents (record series such as ‘refurbishment project records’ ‘client case file records’ ‘consultation records’ etc.) rather than to individual document types (‘project initiation documents’ ‘research reports’ ‘meeting minutes’ etc.)

• There is no way of using content types for both record series aggregations and for document types (because a document can only belong to one content type at a time).

Question from the audience about document sets
I was asked whether ‘document sets’ (a new kind of content type available within SharePoint 2010) could be used to aggregate different content types. I said that although document sets do aggregate content types, I did not think they would not work for records management purposes.

Document sets were intended by Microsoft for use with compound documents (such as ‘health and safety manuals’).

In theory document set content types could allow a records manager to:
• define a document set content type for a record series (for example ‘refurbishment projects’)
• enable content types to be used within instances of that document set (for example ‘architectural drawings’ ‘contracts’ ‘contractual correspondence’ ‘meeting minutes’).

The problem is that if a colleague uploads a document to a ‘refurbishment project’ document set and gives it the content type ‘meeting minutes’ then it would take its retention rule from the information management policy defined for the ‘meeting minute’ content type and not from the policy defined for the ‘refurbishment project’ document set content type.

Facilitated discussion on SharePoint governance
Earlier in the day I had the pleasure of facilitating a round table discussion about SharePoint governance. Two intranet managers told me that their organisations had decided to apply strong governance arrangements in the site collections used for their SharePoint intranet, but to be much less controlled in the collaboration environment, where team sites would be given to local teams on demand.

We talked in the group about the ‘Pontius Pilate’ attitude to SharePoint of some IT departments who treated SharePoint as though it was a tool set. They provide team sites on demand, and when it grows out of control they say it is the fault of the business for not governing it properly.

Someone else in the group said that the trouble with governance is that at first it doesn’t seem like a big issue, you have a system that you want people to use, there are not many users yet, and there is not much content yet. Later on, when the system has grown and evolved you realise the need for governance, but trying to put governance in place is much more difficult (like locking the stable door after the horse has bolted).

I said that governance in SharePoint is primarily about taking away powers (for example the power of site owners to create sub sites, and the power of site owners to create content types). There is a resource implication to taking away those powers. You need to give people a quick way of obtaining new sites and new content types where appropriate. And that means having people in place who have the ability to judge whether or not the new site or content type is necessary and appropriate. If requests for new sites/content types are granted on demand then you have not added any value.