Innovation and Complex IT Projects

The recent troubles in DPI with the TRELIS system are symptomatic of the approach that is usually taken to the development of complex IT projects. However we should not be too quick to blame the key stakeholders who led the development of this project on behalf of the customer proxy, the government, who represented the true customer being ourselves, the people of Western Australia.

I want to try and shed some light on the real-life complexities, problems, conundrums and deficiencies that tend to lead large, complex, costly and highly-politicised projects off track almost from the very start and offer some solutions provided by experts in this field. More importantly and in a broader context, I want to explore how the development of complex IT projects and associated software relates the state of our cultural, intellectual and creative capital in our state of Western Australia.

The problems that have been experienced on TRELIS and other large, complex IT software systems development in government are very similar to those often experienced in commercial organisations. This point leads us to conclude that such challenges are not due to government department culture but are due to more-general causes.

I say challenges rather than failures deliberately because these kinds of projects are seldom failures and are often unsung, large-scale successes. TRELIS may indeed be one of these because the system does work, generally speaking, in spite of the Auditor General finding, "Poor specification of business requirements and software development problems resulted in TRELIS being two years behind schedule when it went ‘live’ on 6 July 2004, " (*ag).

Granted there are problems but any "first" is going to experience these - such is the nature of new development of any kind. The fact that, "An approved business case for TRELIS could not be located" is irrelevant because the system upgrade was necessary and the fact that, "Funding for TRELIS was provided on an incremental basis rather than total funding to match the requirements of the project and the level of project management, oversight and quality control were inadequate for a project of this size" is heavily misleading.

The requirements were incomplete because it is not possible to know them prior to the project undertaking requirements analysis, a first cut of which is produced at the start of the project but the process of requirements discovery and elaboration of hidden and misunderstood requirements is ongoing throughout the lifetime of the project, declining as the project reaches completion.

Without complete requirements (as noted, impossible to produce for most systems) estimation and hence budgeting and funding cannot be perfectly accurate. For projects of this scale of magnitude and complexity to go over budget and over time is not only defensible it is to be expected. But we can do better.

The book Waltzing with Bears by DeMarco and Lister is about managing risk on software projects and is one of several easy-to-read management guides about project management and estimation that go beyond the meaningless practice of tabulating fixed estimates for each task in the work breakdown structure at the start of the project and then pretending these reflect delivery expectations when in reality they are merely guesses and nothing more.

The title of Death March by Ed Yourdon says much in itself and talks about the symptoms of pathological management that is reasonably prevalent in the software industry, at the same time presenting practical solutions to dealing with these circumstances. The Guide to the Software Engineering Body of Knowledge, or SWEBOK, is not well known to many practitioners. Both traditional and contemporary approaches to software project management, from Rational Unified Process to Agile approaches that include Scrum, Extreme Programming and Feature Driven Development, are often ignored or misunderstood in their application.

The report entitled Challenges of Complex IT Projects jointly authored by the British Computer Society and The Royal Society of Engineering goes straight to the point of the many problems that beset complex IT projects. As I note in a submission to the National Innovation Review, "there is a broad reluctance to accept that complex IT projects have many similarities with major engineering projects and would benefit from greater application of well established engineering and project management procedures. For example, the importance of risk management is poorly understood and the significance of systems architecture is not appreciated."

The report goes on to point the general direction that needs to be taken in order to remedy the situation facing organisations which undertake complex IT projects. Two key recommendations are: 1) There is an urgent need to promote the adoption of best practice amongst IT practitioners and their customers; and 2) Basic research into complexity and associated issues is required to enable the effective development of complex, globally distributed systems.

In Perth, perhaps a reflection of the pioneering approach taken by the explorers of Western Australia, and reflected in many wonderful technology success stories that are little known but should be much celebrated, cultural change towards openness to innovation and willingness to change is rare. Companies such as FormSys, Maptek, QPSX, ERG, Asgard, Sanford Securities, JDV, L3-Nautronix and many others have each had their measure of success in their respective application areas.

There is so much opportunity in mining and resources, communications, defence, utilities, government and enterprise IT systems development where Perth companies can carve out an outstanding niche for themselves or join the list of national and global leaders. For example, strong and capable leadership is needed for us to to take advantage of benefits that could accrue to our state due to all six of the Collins class submarine fleet being based here, the incredible development of industry at Australian Marine Complex and Technology Park, to support research and development work in subsea engineering, communications, other marine, acoustic and defence technologies.

The current and future benefits of the iVec supercomputing centre, the global radio astronomy project known as the Square Kilometre Array or SKA that is looking more likely to be based in our state, continuing massive investments in the mining and resources, associated infrastructure, knowledge base and information technologies are potentially enormous and could leave an unparalleled legacy as a benefit of the extended boom we have experienced in mining and resources.

Beyond the boom, the development of advanced Information and Communications Technologies, known collectively as ICT, will underwrite the future growth and well-being of Western Australia where Rabbitnomics dictates that mining alone will not. Earlier this year, I was privileged to be a part of the Australian Software Engineering Conference that was held for the first time in Perth in the 20 year history of the conference. Having finally started to build the bridges, it is vital that business development extend to the commercial exploitation of the creative capital we have in abundance in the fields of ICT in our state.

As part of a much-needed debate about the cultural face of Perth, the abundance of creative capital and deficit of innovation, FORM brought Charles Landry to our city, who wrote that, "A great city must be much more than wealthy. It must have a culture rich in passion, compassion, art and innovation."(*cl) FORM and the debate over creative capital, culture and ideas was naturally dominated by architects, psychologists, artists and politicians, with little input from scientists, engineers and other technologists.

It is quite natural that social scientists and others focused on cultural and social issues should frame and dominate the debate about the composition and geography of Perth, its relationship to our beautiful surrounds, and our place in the environment as members of a community. However those across the Maginot Line that fortifies the supposed partition of social from physical sciences have a role to play. The physical sciences, engineering and technological arts have a closer relationship with their artistic and literary brethren in commerce, law and design than many realise.

Creativity is rife throughout scientific endeavours of discovery and complements the rigours of design. It is just so that structural engineers and software engineers share the terminology of requirements engineering, specification, design and concreting or concretion, in the former case the pouring of concrete foundations and more elaborate vertical structures in order to satisfy the architectural blueprints, and in the latter case of software the artifact written by a coder to implement the software design.

In both cases, the term architect is used appropriately to describe the role of the designer who conceptualises what is to be built. The term software architect was borrowed into the field of software engineering by a body of workers who constructed software design patterns based on the work of Christopher Alexander, a famous architect known for his theories of design and in particular his seminal book on planning called A Pattern Language: Towns, Buildings, Construction.

In the software engineering community, the seminal book on design patterns is Design Patterns: Elements of Reusable Object-Oriented Software which follows the same pattern templates or scripts as laid down by Alexander. The parallels are obvious and I use this as one example of the close, even symbiotic, relationship between concepts and their practical realisation in fields as diverse as architecture and software development.

Likewise, similar arguments can be applied across the full spectrum of creative undertakings by physicists, biologists, mathematicians and engineers of all stripes who mix the spark of creativity with their foundational sciences and practical technologies in order that an innovative outcome shall emerge.

It has been suggested at various times that our state and our universities should specialise narrowly in fields that are central to or that support the mining and resources industry, in order to leverage our competitive advantage in this area due to the ample resources gifted by nature. However I contend that such a view is short-sighted and counter productive because the wider society would suffer as a consequence.

Besides, we need to embrace a strategy that takes us beyond the boom and we know that knowledge industries including ICT, biotech, subsea, marine and defence present some of the best opportunities for succeeding in a crowded, global marketplace for products and ideas.

(*ag) http://www.audit.wa.gov.au/reports/report2006_01.html
(*cl) http://www.thewest.com.au/default.aspx?MenuID=77&ContentID=23433