Writing Documents – Part 1

This is the first of a series of posts focusing on creating documents and presentations.  It has been triggered by reviewing several documents of late that needed a fair amount of remediation and various people asking me how do I approach building documents & presentations.

There is no secret or magic formula to creating a good document, the most basic thing you need is “Practice” combined with constructive feedback from your peers, managers and document recipients.  Over the years I have been fortunate enough to have had the opportunity to produce probably more documents and presentations than is considered healthy :).  But more importantly I have had a number of excellent peers, managers and customers who have been willing to offer advice and teach me the tricks in creating good documents.  Over this time I have adopted my own style and also developed a few tricks of my own to help in creating documents.

I deliberately use the word “Creating” rather than “writing”.  Whether it is a status report, solution design, business requirement or novel, the process of producing a document is a creative one.  Regardless of the content & purpose you are essentially telling a story and like all stories it needs to be a good one in order to keep your audiences attention.  And more importantly in the business world you are being paid to write the document so the “story” must justify and reflect the cost of production.

So how do you start creating a document?

Forget about templates and styling for the moment (we will deal with these in a later post) and think about how to begin.  Everyone has their own way of approaching a document or presentation.  I use the following graphic as my basis for approaching every document I create.

 

Coming from a technical background I have always found the following techniques work best for me, everyone has their own approach and style but they will probably use one or more of the following;

Output Driven

When people think of outputs in relation to documents or presentations they often think only of the document as the output.  Depending on the purpose of the document the outputs will be contained within the content, the document is only the container.  So for example;

• A Solution Design would have outputs such as “the design”, “assumptions”, “required resources”, “next steps”, etc.
• A feasibility study would have outputs such as “options”, “cost benefit analysis”, “recommendations”, etc.
• A review document would have outputs such as “current issues”, “conclusions”, etc.

Knowing what the end-state needs to be is useful in helping you focus on the structure and message to be conveyed by the document using the available inputs.

Story Boards

As I mentioned earlier, every document must tell a story.  The concept of story-boarding is not only for film producers.  Take time to sit in front of a whiteboard and draw out the “flow” of the document.   This is a particularly good technique for creating presentations.  Before you even sit at a screen and start up Word, PowerPoint, Pages, Keynote stand back and envisage the flow of the story.  Get a feel for how it will flow and how many chapters / slides are necessary to convey the message in a precise unambiguous manner.

Sketch out the flow and “talk yourself through it”.  Working at this high level allows you to focus on the big picture and visualize how the final product will look and how the message will be conveyed.

Top Down Approach

This is an approach normally associated with computer programming and to be honest I have adopted “Jackson Structured Programming Methodology” (showing my age here) to work for me in creating documents, presentations & plans.  Virtually anything that needs to be organized can be done using this approach.

The value of this approach is it allows you to work at a high level and work down to the details without any of the details assuming a higher degree of importance than the others.  This allows you to ignore areas that you are not sure of at that point in time and focus on the bigger picture and story flow.  Using this approach in combination with a story board is a great way to get the story worked out and to build out the Chapters, Sections, Slides, etc.

In the next post I will examine influences, structure and content.

Delivery, Delivery, Delivery

What is it with a significant number of organisations that impairs their ability to deliver software to production in a timely manner with a high degree of quality?

Most delivery issues can be traced back to lack of decision making capability, too many changes being released at once and a failure in existing (or poor) organization & processes.

The best means of building a reliable software delivery organization is to get in place the pieces necessary to deliver one change to production in a timely, secure and organized manner.  If you can figure out how to do it for one simple change, and the process utilised ticks all the necessary boxes for decision making, control and speed then you have a roadmap for scaling up to delivering more functional changes & fixes at the same speed and with the same reliability.

Consider the following when defining a delivery process;

• Define requirements as “simple unambiguous testable propositions”
• Design solutions that can be divided into simple single function components
• Build the software in a manner that is robust, fault tolerant and efficient. Focus on test driven development approaches
• Deliver the software to a robust QA / UAT environment that is an accurate representation of production
• Use automated testing tools where possible
• Use automated delivery tools where possible

Having a rapid process controlled delivery model allows for;

• Delivering small
• Delivering often
• Changes in decisions
• Fast response to critical business issues
• Agility in the analyse-design-build model

Focus on building a robust production line for delivering change to production systems.  Invest in the tooling and environments needed to ensure consistency and quality.  Make it easy and reliable to deliver change.  Once this is achieved press the accelerator and speed up to the point that best suits your organization.

 

Migrations should be Easy!

Migration of Data from a legacy application to a new application is probably one of the most fraught activities companies undertake.  Most companies will view this activity as the biggest risk in moving to a more modern system and a significant number will drag legacy data structures into a new application making it behave, look and feel “just like the old one”.  Just why this happens is open to debate but in many cases it is down to;

• Lack of a strategy outlining primary reasons for bringing data over
• Lack of a well thought out process structure
• Fear of leaving something important behind
• Uncertainty in what has to be moved

In its simplest form migration is comprised of;

• Data Structure Assessment
• Data Quality Assessment
• Data Cleanup
• Extraction
• Transformation
• Data Load
• Reporting
• Historical Records

Basic Principles

Before even getting into the details of the data the first deliverable should be a comprehensive report outlining strategy, detailed set of processes, RACI matrix and a broad set of timelines.  By defining in detail the end to end process to be executed, everyone involved can see what the target is (and is not), who is responsible for what and when the activities are to commence / finish.  The surprising thing is that most companies either fail to produce such a document as a first step or re-invent the approach from scratch each time a migration is approached.

Yes there will always be items that are specific to your migration but in principle every migration can be broken into a discrete set of steps that vary little from project to project.  The list of activities above can be considered level 1, each of these can be broken down into 20 level 2 activities and each of these should again be divided into 5-10 level 3 activities.  The purpose is to define the migration process to such a low level of activities that each one in itself is no longer seen as a “black hole” or “difficult to envisage”.

In addition the migration approach should be viewed and managed as an “automated” and “repeatable” set of activities.   The toolsets used during migration should allow the insertion of data fixes, data moves, etc as SQL or command scripts which are executed whenever the process is run.  This allows the team to build up the “Migration Instruction Path Repository” that can be signed off in UAT and then repeated in production without people “remembering” to perform X at a specific point in time.

Data Structure Assessment

Once the approach and process is signed off the detailed analysis can start.  Many projects that I have witnessed begin this process by looking at the source data, figuring out what is there and then using this to dictate what is dragged into the target system.  This is a dangerous practise as source data will then drive target system requirements and force changes to the target system to cater for source system processing models.  The right approach is to actually define the target system data requirements, define a staging database and then only take what is necessary from the source to satisfy the needs.  I have come across projects where people trawl through screens looking for data the users currently see / interact with to define “what is required” – this leads to unnecessary levels of change in the target system to accommodate old style data structures and perceived user needs.

A “staging Database” should always be used as the transition point for the data – you should never go directly from Source to Target.  Ideally the staging database is a de-normalised entity model of the business domain and should not reflect either the source or target physical database structures.  The purpose of a staging database is to capture the “Critical Data” needed to build the required data model in the target environment.

Data Quality & Cleanup

Data Quality and Cleanup are essentially activities that firmly belong to the Source Data system.  Before any attempt is made to extract data you should make sure it is valid.  Any inconsistencies or data errors should be fixed inside the source database.  At a minimum any quality issues that cannot be fixed must be logged and audited for final verification.  As a rule fixes to data quality should not be attempted in the staging database or in the Target database, particularly fixes to financial data.  What was in the source must be reflected in the target otherwise a migration is difficult to verify and signoff.  It is the responsibility of the Source Data owners to provide as clean a data set as possible for extraction to the staging database formats.

Data quality should be verified using a Data Quality Toolset which allows quality conditions to be defined and executed automatically.  This allows the rapid creation of data quality conditions in a non technical format which are then automatically translated into standard SQL’s and executed automatically.

A similar type of toolset can be used to build up data cleaning rules.

It is also advisable to have a set of automated quality checks which you can run over the staging and target databases.

Extraction and Transformation

The extraction process will have a degree of transformation involved as it must make the source data fit the staging database model.  Transformation of individual data items to new formats (e.g. dates) or values (e.g. codes, etc) can be done either inside the extraction process or as a separate transformation activity focused on the staging database.

The data provided to the staging database should be as clean as possible.  As mentioned before, no fixes to bad data should be made inside the staging database, it is a transition point only.

Only extract data for entities that will be loaded into the target database.  It is the responsibility of the extraction process to filter out anything that is not required (cancelled policies, old & unused clients, etc)

Data Load

The loading of data from the staging database is a process where the target data is created and processing nuances of the target system are built to allow this data to be processed going forward.  No transformation or data change activities should be part of the Load process.

Reporting

Apart from testing the migrated data to prove the transfer of data has worked the single most important task in any migration project is to prove that the data in the target database is a like-for-like record of the extracted data from the source database.  Reporting in this context should be performed as follows;

• Define a comprehensive set of value summaries (hash totals, counts, financial, etc) to be tracked
• Run the reports to produce these summaries over the source database, staging database and target database
• Compare the source and staging summaries.  If there are discrepancies then either find the issue and correct them or document the reason for their existence
• Compare the staging and target summaries.  If there are discrepancies then either find the issue and correct them or document the reason for their existence
• Compare the source and target summaries.  Any differences should be accounted for in the documentation produced by the previous comparisons.

It is important to conduct the comparison in this way as direct comparison of Source & Target can be difficult to reconcile if major data structure transformations have occurred or any “fixing has been executed as part of the extraction process.

As with the data quality check, having a good reporting toolset that allows easy definition of summary totals, automated build of reporting assets, automated execution in a repeatable manner and result deviation logging is essential to making this activity easy to manage.

History

In every migration there is always the vexing question of how much history to take and how much history to build into the target database.  Firstly, history is expensive to create so at a minimum only historical data that is legally mandated and or critical to the management / processing of the migrated data should be considered.

Building out a full set of history inside the target database in a manner that resembles full processing from day one of a data entity’s life is extremely expensive, difficult and very rarely worth the effort.  From experience this degree of history generation (if possible) can more than double a migration projects cost and timeline.

However users will need some access to historical events for normal day to day business needs.  This can be solved in a number of ways.

1. Keeping the old system available for enquiry only access is the easiest solution.  However this approach can incur significant costs and management effort particularly for commercial software licences and out of data hardware retention.  This approach is not generally recommended.
2. Putting all the old data in the target database in “static data tables” and building enquiry facilities is a favoured approach.  However this also has its disadvantages in that the new system’s DB becomes cluttered with data it does not use and the temptation is always there to drag this “source” data deeper into the new systems engines corrupting its distance from the source system heritage.  An evolution of this approach is to dump the old data into a separate database retaining its structure and building a simple enquiry facility.
3. A more flexible approach would be to gather the data in entity focused XML documents and build a simple intranet-site to display the information in the manner required by the users.  This approach makes it easier to display the data in different format’s suitable to individual users/departments as the data is  held in a flexible form.  It also makes it easier to print or export full copies of data entities as required.

and Finally

Every migration will have its own nuances, variations and special needs, however the overall approach and execution differs little from project to project.  Approaching the task with the following in mind will make migrations easier to plan, manage and execute;

• Document the strategy and define the low level processes to be executed so that everyone knows what, why, when and who
• Automate as much as possible and mandate “assured repeatability” is enforced in all activities
• Use toolsets to help ensure repeatability, documentation & decision management
• Do not let the source data model dictate the target state
• Clean up at the source
• Be frugal, import to the target database only what is critical to the success of the business

Being Agile in a Waterfall World (Part 2)

… continued from post … “Being Agile in a Waterfall World (Part 1)”

So how do we work in an agile manner in our waterfall world?

While people like the adaptive nature and speed to delivery of an agile approach in software deployment the one area that raises real concern in big projects is that the Iterative approach will lead invariably lead to “Scope Creep” and attitudes of;

• “I do not know what I want but will tell you when I see it”
• OR – “This is my chance to get everything I always wanted”
• OR – Continuously “tweaking” the requirement and never getting to a conclusion

This is a valid concern where users are not trained/encouraged  to focus on what is really needed and/or the IT Delivery Staff are not empowered to push back.  It can and does lead to all sorts of problems with scope, deadlines, costs, etc.

What we really need is an approach that combines the accountability scope control, and predictability of a waterfall model, with the agility and adaptability of the agile model.  Lets call the approach the Agile-Waterfall and define a set of principles that we would like to be present to govern our projects;

Accurate Project Scoping is absolutely critical for determining commercial & delivery agreements between customer and supplier/IT Dept.  So our Agile-Waterfall philosophy should emphasis the following;

1. Define the minimum necessary requirements to satisfy the business need upfront (our “must haves”)
2. Capture the remaining requirements into a “would be nice to have” grouping
3. Organise the project to deliver based on business architecture with realistic functional area sequencing
4. Promote the ethic of requesting & building just enough to work effectively
5. Measure and report on progress by a deliverable which is a working component not document artifacts
6. Utilise Agile principles within the delivery of these components but keep the waterfall control elements to monitor spend of both time & money and also scope creep
7. Build team-units that have all the requisite skills to take ownership of delivery of individual components – from requirement to working software
8. Build an environment that encourages people getting invested in the project
9. Involve the users actively in the execution of delivery

The following diagram is a representation of how a project could be represented to allow us to be Agile within the confines of our discipline scope

We still perform a “Discovery” exercise to gather the vision of the project.  The level of detail gathered should be enough to Define Scope and in a Fixed Price Contract define Time and Cost.

The Project is then organized around logical blocks of work that are executed in a sequence that makes business sense.  Foundational elements such as major DB modifications, Product Definition & Parameterization Extensions, etc are done first to allow the business components to progress without too much inter dependency.  Foundational components also refer to big requirements that span many business areas – although these should be kept to a minimum.  High level approaches & guidelines need to be defined to allow the teams to work in a consistent manner.

The requirements are divided into blocks of work that focus on specific business areas and are sized to allow delivery within a maximum targeted sprint time frame, i.e. 2, 3 or four weeks each.  The reporting of progress to the project sponsors should be at the Business Component Completion level and no lower

Each business component should  have a maximum manday effort and cost allowance to achieve its target.  It is the responsibility of the team assigned AND the requirement(s) owner to ensure these limits are not exceeded.  Standard Change Control procedures can of course be implemented to cater for those situations where more time is necessary due to under estimation or unexpected complexity.  The teams must build the bare minimum necessary to meet the “Needed” requirement; excess features & additional requirements can and should be pushed to subsequent phases.  This encourages users to actively participate as they know the delivery organization is only saying “Not Now” rather than “No” to their requests.  The business component’s rollup  into the deliverable “Phase of Work” for the formal SIT & UAT activities that are carried out once all the components are completed.  There should be a Test Support & QA activity running in Parallel to sign-off on the individual business components as they are completed.

Each Business Component can be viewed as follows;

Each Requirement goes through an iterative process to first deliver what is necessary and then within reason/scope deliver what is additional.  Each Business Component can be comprised of one or many requirements relating to a particular area.  A business component’s team will deliver;

• Working and tested software to the project phase
• Sufficient documentation to justify the deliverable and its approval by the user community
• And Optionally they can deliver excess feature requests & requirements back to the master list for inclusion in later phases.

Each business component team must comprise people with the following skills Business Analysts, Developers, Testers and Users.  One member of the delivery team will assume the lead role for monitoring and reporting on progress and issues.

Consider the Principles contained in the Agile Manifesto below (also see http://agilemanifesto.org/principles.html)

1. The highest priority is to satisfy the customer through early and continuous delivery of valuable software
2. Welcome changing requirements, even late in development
3. Deliver working software frequently
4. Business Users and Development Teams must work together daily throughout the project
5. Build projects around motivated individuals with the proper support and environment and trust them to get the job done
6. The most effective and efficient way of conveying information to and within a development team is face-to-face conversation
7. Working software is the primary measure of progress
8. Agile processes promote a sustainable development cycle
9. Continuous attention to technical excellence and good design enhances quality
10. Simplicity – the art of maximising the amount of work not done
11. The best architectures, requirements and designs emerge from self organising teams
12. At regular intervals the team reflects on how to become more effective and changes accordingly

By viewing how a project is organized and enforcing the scoping boundaries (for budget, effort and schedule control) I believe it is possible to practice each of the above principles for a  Business Component delivery structure and work in an Agile manner inside a Waterfall focused project environment.

Some additional thoughts on various approaches and practices that can assist us with moving to Agile in a Waterfall world;

• Get the business community socially invested and involved early and keep them involved throughout the project – To get maximum benefits of using Agile principles in a project it is advisable to setup and staff a Model Office at project commencement.  A Model Office is composed of a group of the best users who have an intimate understanding of the company and are empowered to make decisions for the betterment of the organisation.  The model office is the liaison between IT Delivery and the user community and from the IT perspective they are the user community.  The Model Office manager should in theory report to the CEO or COO.  I will deal with the concept, value and structure of a model office in a later post.

• The Model Office should be comprised of people whom the company finds difficult to extract from the day-to-day operations due to there importance.  If you are going to spend $Xm on a project surely you want the best people to see the money is directed at the right requirements

• Promote the following basic Design and Coding principles.  These are development language agnostic, I will also discuss each of these in more detail in a later post;

Test Driven Development (TDD) & Iterative Coding Write your program/object in steps and making sure each part works before moving on to the next part rather than knocking out 2000 lines of code and then trying to compile and test it.  This is a fundamental part of TDD.

Iterative Development Get the basic requirements finished first for assessment, then add in any additional features or requirements which are not beyond the original agreed scope.

Use of Single Function Design provides discreet blocks of code, programs, objects, methods that do “One Thing”.  These blocks of code either work or not, they are easy to test and then move on to the next one.  You then combine these pre-tested units into more complex structures where all you are doing is testing the combination not the core logic.  This leads to faster deliveries, higher quality code and a higher degree of re usability.

DRY (Don’t Repeat Yourself) is a coding model that promotes avoidance of duplication.  It is about expressing information/logic in ONE PLACE and leads to better consistency in code and is easier to maintain.  Be aware though that using the DRY approach does mean you need to refactor code as you progress to remove the duplication created by the introduction of new items.

Data Driven Engine programming is a technique where you build code that reacts differently to the data used to configure it and data passed to it for processing.  Where possible building this type of solution will allow greater flexibility for reuse and more solid code which does not have business logic imbedded in the code, so when change happens you change the control data not the coded logic.  This works best with the “Single Function Design” approach.

Defensive Programming is a technique where you check if something will fail BEFORE executing it and make a decision on corrective action or controlled degradation of the application.  (e.g. checking if divisor C is 0 before executing A = B / C or if program/object XYZ exists before using it)  Do not confuse defensive programming with checking the result of an operation AFTER execution.  That is “Reactive Programming”.

Structured Programming Techniques alludes to correct design of code before you sit down and write it and the proper use of structured elements within the coding model to allow for ease of maintenance and understandability.

And finally for any project – Test iteratively and Test Often – From a testing perspective let the users see and play with what has been built as soon as possible.  It does not have to be a full blown UAT style exercise but at least get confirmation that the deliverable is OK in the eyes of those who requested it.  Finding issues as close to the point of development as possible reduces cost in changes / bug-fixing.  If possible introduce a regime of “Continuous testing”.

When is it time to stop evolving and re-build from the ground up?

For most major software applications, the companies that build them get trapped into a never ending cycle of enhance, extend, modify …… and fix.  Due to the nature of tight release schedules and increasing complexity the practice of “Refactoring” is either ignored or banned outright.  Move forward 10-15 years and what was a great product starts to look like a patchwork quilt of good ideas and band-aids to keep these ideas working together.

So back to the question raised above ….. when is it time to rebuild from scratch?

Rebuilding from scratch is not about throwing away everything you have done but it is about realizing what is good, what is bad and that there are better ways to do things because of technology changes, customer needs, hindsight, etc.

Personally I would advocate fully rewriting the core code base at least once every 7-10 years and always whenever you are changing technology platforms (either hardware or software).

To make this effective you need to build a “Modernization” strategy and philosophy into your product from the beginning.  This will make it easier to preserve good ideas & code and throw away outdated ones.  It will also make you think about how to bring your customers with you as you evolve in small steps and revolutionize in big ones.

Off-shoring – Staff Augmentation v’s Sub-contracting

Most organizations today are using or planning to utilise lower cost centres to fulfill more of its project needs both on a cost reduction and execution basis.

The cost reduction ethics of utilising people in economically less developed countries is a well understood model although the rationale of hoping that cheaper will be better is usually ill conceived and badly thought through, the biggest folly in this area is generally people are so fixated on cost per hour they over-look off-shore effort (mandays) per deliverable which in a lot of cases negates any cost savings.  However this article is not about the rights/wrongs of using lower cost labour but it is worth considering the following “A competent software designer/developer whether on shore or off shore is an individual of value and if not respected as such will look for a higher paymaster once a sufficient degree of skill is reached and true value is not being properly rewarded and/or utilised”.

The execution basis rationale is based on the belief that a larger population in economically less developed countries provide a higher ratio of people to design and develop software.  This in itself is a valid assumption if normal market demand applied, however when multinationals descend on a country to snap up an eager and educated workforce, within a short period of time the laws of supply and demand drive both the costs up and the average skill levels down as people rush to cash in on the demand side and the natural-talent % of the work force is reduced by the sheer numbers hoping to get trained in order to have a decent prospect of employment with a respected company.

The software deployment industry are masters at gathering up any available talent and deluding themselves that numbers through the door “i.e. backsides on seats” equals proportionate gains in productivity and throughput.

It is easy to find stories of failure and distress at off-shoring efforts, but more difficult to find real success stories.  By “real success” I mean a repeated demonstration of delivery capability by a core team that has been together for more than 2 years and are masters at delivering either components of or whole projects in a remote capacity.

And this brings us to the nature of this topic, we as an industry need to drop our staff augmentation (i.e. body shopping) mentality for off-shore usage and focus on a modern, professional and sustainable model more akin to the sub-contracting models utilised by manufacturing, logistics, distribution and other industries.

Regardless of the country selected as your point of off-shoring if you take a staff augmentation approach you are looking purely at cost reduction and trying to fill in gaps in resourcing, reporting on savings in cost-per-hour and then wondering why overall cost per deliverable has not decreased.  What usually happens is that the team as a whole fail to gel together and people working remotely never get properly invested in the project.

A sub-contracting approach is one which allows an off shoring model to operate in a more productive manner encouraging the staff to build real careers within a company.

So what do I consider Sub-contracting to be within the confines of a software development environment.

First of all lets define what it is not;

• It is not about grabbing a few programmers to do the coding

People need to understand and be connected to what they are doing. Just dumping technical specifications over the wall gives a team no real satisfaction in deliverables and also (and more crucially) does not promote an environment where people have real career paths and motivation to stretch themselves.  This is one of the reasons for high attrition rates in off-shoring organisations.

• It is not about controlling all aspects of the activities from the on-shore project office

There is a high percentage of project managers that do not trust what they do not fully control.  When you push work off-shore there has to be an implicit degree of mutual trust on both sides.  Trying to micro-manage a remote team leads to conflict and a loss of motivation in the team members and a belief that they are not fully empowered to deliver to the best of their abilities.

• It is not about the lowest cost

Off-shoring should be about building an economical and sustainable delivery model where the competencies off-shore are the equal of those on shore.  You must be able to charge realistic rates for off-shore staff in order to maintain a reward structure that keeps people in your organisation for more than 5 years.  And by reward this means a mixture of monetary, experience, career and environment incentives.  People will always leave because of money, but it is much more difficult to leave an environment that you find rewarding in social, educational and career progression perspectives.

I believe Sub-Contracting is best defined as “Employing a team outside one’s own organisation to deliver one or more components of the overall program/project”

Whether the Sub-Contracted party is internal or external to the organisation of the Sub-Contractor there is a core set of values and rules that should be followed in order to make the engagement (i.e. contract) work in a successful & professional manner (this is particularly true in the IT industry).

The sub-contractor manager or business unit owner is essentially running a company and should behave as such;

• They must invest in training their staff to a level that is equal to but hopefully better than their on-shore colleagues
• They should be firm in their approach to what will be accepted as a contract and be forceful in negotiating the conditions under which an engagement is to be executed
• They should be involved in determining how big a task is and more crucially how many resources and  what competencies are required
• They should always be in control of agreeing delivery schedules

Everything needs be treated as a product and the sub-contractor should be fully aware of the “Products he is selling”.  For example a sub-contractor could have a range of project models that can be bought off the shelf.  1,000 Manday development Model, 5,000 manday model, Migration model, etc..  Each model can be tweaked to suit the particular engagement but cannot be fundamentally changed because the on-shore Project Manager and/or team do not agree with it.  Each product should have a well defined and understood cost profile.

The on-shore sub-contractee should be aware that if a sub-contractor says it will take X to deliver a component then they have no right to object and force their own beliefs on the sub-contractor’s workforce.

Sub-contracting is a bought and paid for service, if the purchaser cannot adopt your style of working then let him find another provider.  Your reputation as a sub-contractor will be based on a proven track record of deliveries and a high quality of all deliverables.  You can only build this reputation by being in full control of how and what you deliver.

A formal contract should be in place governing the responsibilities of the Sub-contractor to the Sub-Contractee

• Every engagement should be documented via a Statement of Work with requisite SLA’s on delivery schedules, quality and change management.
• Work to be Sub-contracted should described as a deliverable which can be designed, built and tested off-shore before delivery to the on-shore project stream.

The interactions between the parties should consist of the contract negotiation, agreed milestones/checkpoints, agreed inputs and outputs, agreed governance and QA and above all agreement of the sub-contracting party NOT to interfere in the manner in which the sub-contractor carries out the engagement.

Just as the on-shore team report to the client, the off-shore PM / Team Leads must engagement with their on-shore counterparts in the same professional manner.  Trust in something that is hard to earn and easy to lose.  Professional communication and management approaches help in building trust.

In order to build up a competent and professional organisation you as a sub-contracting manager must insist on delivering components of a project from design, coding through to testing.  You need your people to be engaged and invested in what they are doing, only by offering a full development lifecycle will people feel that they own the deliverable and that there is a well defined set of careers paths within your organisation

Off-shoring using a sub-contracting model can and does work.  However to be really successful you must strive to be perceived as a provider of solutions and not a provider of people.

Being Agile in a Waterfall World (Part 1)

Is it possible to use Agile principles when deploying & maintaining critical solutions where projects have durations measured in a year or years and may involve significant number of interfaces and possibly a migration of legacy data?

What if the application is in a procedural language such as COBOL, RPG, etc?

What if the project is “fixed price”?

There is a tendency in such major projects to treat the term “Agile” with some or all of the following;

• a high degree of mistrust
• a sharp intake of breath and shaking of heads
• objections such as “won’t work here”, “does not suit our approach”, “only works for web languages” and “agile is for web startup’s and small non-critical projects”
• Cannot work for projects with a single point of delivery

For the sake of clarity and to avoid misunderstanding, I am not advocating the use of an Agile approach as a silver bullet to cure the ills of critical,  complex, long duration projects.  I do however believe there are certain Agile principles and approaches that if utilised could have a positive impact on these projects.

There are limits to being truly agile in the world of major projects and it is rarely feasible to go live on an iterative basis when you are migrating a 30-year-old portfolio into a new application, replacing a core ERP solution or even introducing a new component to the current architecture landscape.  However there is no reason you cannot practise some Agile principles within the boundaries of the project.

The most surprising aspect I have found when the Agile suggestion is raised is that it is not generally the business users who raise objections, it is the IT Departments and IT Vendors.

Before we think about the details of agile principles and approaches, lets be honest with ourselves, current waterfall approaches to projects are not working effectively! Project participants regularly complain about some or all of the following;

• Requirements are unknown, constantly evolving or in such an abundance no-one can see the wood for the trees
• Users are either disengaged and fight against change or incorrectly engaged in that they restrict the development & design staff from delivering real change through a misunderstanding of the projects objectives for the business
• Documentation is treated as a critical artefact not as a means to generating working software
• People rely on the “Process” of project execution rather than the reality of what is required to be done
• People are micro managed, project plans for 18 month projects are maintained at a level showing 2 and 3 day tasks
• No one has fully committed themselves to the project
• The project has no single owner, Project plans for 18 month projects are maintained at a level showing 2 and 3 day tasks
• A price was agreed before anyone knew the true complexity of what was to be delivered, this is usually accompanied by a specific delivery date that is sacred and cannot be moved
• The “I/We want” continually overrides “Business Needs”
• Development teams spend an inordinate amount of energy building the vision and then arguing with users about why something is not working the way everyone thought it would
• Delivery Managers spend equally long periods of time managing everything to a 1000+ line plan, arguing with clients about why things are done in a particular manner, juggling resources and at times not utilising the best skills of the team members
• Project Committees mistrust the vendors, do not deliver proper support to projects and when things go awry demand daily status reports that basically prove the team can do status reports but little else
• Client Management & Staff ignore the fact they are implementing a “new” solution and instead of  advocating internal change to make best use of this solution they try to fundamentally change the new solution to work to the “old” way
• Everyone ignores the old staples of
  1. Fast (Time), Good (Quality), Cheap (Cost) – you can only have two
  2. 80 / 20 Rule
  3. Keep it Simple

It is a wonder how we ever manage to deliver projects in such circumstances, actually its more of a wonder why people even bother trying.

There is a fable that is sometimes used to start discussions on an agile approach;

A Pig and a Chicken are walking down the road.  The Chicken says, “Hey Pig, I was thinking we should open a restaurant!”.  The Pig replies, “Hmm, maybe, what should we call it?”.  The Chicken responds, “How about ‘Ham-n-Eggs?’”.  The Pig thinks for a moment and says, “no thanks, I’d be committed, but you’d only be involved!”.

The fable can be extended by including the Rooster who thinks this is a great idea and wants to manage the restaurant!.

This fable is best explained by the following paragraphs adapted from http://en.wikipedia.org/wiki/The_Chicken_and_the_Pig

“The fable identifies the main project participants, the Pigs who are totally committed to the project, the Chickens who consult and the Roosters who strut around offering uninformed and unhelpful opinions.

A project needs some Chickens and an abundance of Pigs (Roosters are seen as surplus and unproductive).  However given the sacrifice required of being a Pig – fore-swearing all other projects and opportunities – they can be difficult to collect.  Thus the construction of a successful project team must ensure the project has sufficient “pigs” and that they are empowered to drive the project to completion in return for committing to and taking accountability for it”.

The most rewarding and successful projects for participants are ones  where a single person takes ownership & accountability and empowers those below him/her.  In today’s environment our projects are generally run by committee with no one person taking ownership.  Whats even more disheartening is that very few of the team members are empowered or encouraged to take on a personal sense of ownership.   In essence today we have too many Roosters, an over-supply of Chickens and the Pigs are practically extinct!

Consider the following statements;

• Individuals and Interactions over processes and tools
• Working Software over comprehensive documentation
• Customer Collaboration over contract negotiation
• Responding to Change over following a plan

Being agile does not mean you ignore the right hand side of the above list, but it does mean the left hand side takes precedence.

1. Individuals and Interaction get things done not processes.  We are delivering complex software solutions, you cannot define a process that delivers the solution.  You can only define processes that provide an environment which either enables or restricts how effectively people can deliver the solution
2. Working Software, now there’s a term that is rarely used in our status reports.  Demonstrating something works is much more powerful than reporting on it.  Also all documentation produced should be aimed at the final delivery of working software, rather than ticking off a box on a checklist.  Documentation should not be produced for process or methodology’s sake it should be produced because it is necessary to get software working correctly.
3. Customer Collaboration is something we are neglecting in today’s world and it is the fault of all sides.  We spend more time on contracts and tying each other up in knots than on working effectively together.  The sense of partnership has been lost to the goal of winning at all costs.  To customers I would say “if you want the lowest price and to control the delivery process then do not complain about what gets delivered!”.  To vendors I would say “be more upfront and honest with the client and make sure you convey the reality and complexity of the project BEFORE you start” – see an earlier post titled ‘The trouble with Optimism v’s reality’”.
4. Responding to Change is something we are all particularly bad at.  We live in a constantly changing world, stuff happens, decisions made six months ago may no longer be valid, heck decisions made last week can be invalidated by todays information.  Accept change as a fundamental part of the project and be organised in a way so that it is dealt with effectively and not fought against.  Remember a plan is only valid until the first task is executed, the plan is then out of date and only suitable as a wall covering.

Agile is about empowerment of individuals (users & developers), focusing on the right things at the right time,  delivering consistently and more frequently to the user community and improving the quality & relevance of the deliverables.

Even if agile does not significantly shorten the project timeline, using it to meet the following objectives will make the project execution more efficient and less stressful for all parties;

• More efficient requirements discovery and management
• Managing change effectively
• Improving quality of deliverables
• Shorten the time for request-build-review cycles
• Efficient prioritization
• Only Deliver what is necessary
• Building a sense of partnership, co-operation, ownership and trust
• Reduction in Risk

Adopting elements of an Agile approach must also take into account the scale of the projects we undertake.  (e.g. We live in a world  where the team will be dispersed over different locations, we may need to produce additional artefacts to take into account this dispersal, client and vendor will have staff constraints, sometimes a big bang approach is the only possible approach rather than iterative deliveries to the business.)

In the next post I will discuss the Agile Principles and how I believe they can be applied to our Waterfall encased projects

… continued in post … “Being Agile in a Waterfall World (Part 2)”

Laying the Foundations – Figuring out what your business NEED’s rather than what your users think they WANT

It has been called many things “Discovery”, “Gap Study”, “Requirements Analysis”, “Integration Analysis”, “Differentiation Analysis”, etc.  What it tends to be is a whole set of discussions, compromises and disagreements regarding;

• how long the delivery team will be given to uncover the requirements
• how much disruption can be tolerated by the users
• are the project team allowed to talk directly to the business or do they have to use client BA’s as their conduit
• why can’t the project team just look at the old system and figure it out from there?
• will the business adopt the new application or adapt it (i.e. generally this equates to “we are special and have to have something that nobody else needs” … and tends to mess up the base application and drive costs through the roof!)
• what is now in scope that was omitted at the point of contract signing
• we (the client) do not really know what the new products are but want to start the project anyway as we do not want to lose time

And to throw further confusion into the mix; the client users see this major project as a means of getting everything they always wanted, this is compounded by the fact the vendor business analysts are most often not properly empowered to push back on user requests and end up being glorified stenographers.

Every project I have been involved in and heard about goes through the same basic drama ……. now lets sit back and think about this;

• From the clients perspective they have just signed a deal where they will spend $Xm to deploy an application that is going to be used within the organisation for the next 10 years or so

• From the vendors perspective if the project fails or runs into trouble they will be accountable to the client for not getting the right requirements, not steering the client in the optimum direction for best application usage, allowing the project to overrun budget and timelines, etc

For two organisations that are just about to commence on a major project – the first thing that gets discussed is how to shortcut the process, save money, try not to use the most experienced staff and avoid conflict in the lower ranks of users.  This is probably one of the most ludicrous situations we continually find ourselves in.

Implementing a new core application (think business engine) is an opportunity to look at what is really needed and what can be changed for the better.  To succeed you need the commitment drive and experience of the best people in both client and vendor at the start of the project.  You also need to evaluate what is really important as a requirement and what can be pushed out to a later date or dropped completely.

Being prudent with expenditure (resources, time, money) on a project is expected but you should aim to reap savings in later phases based on laying a solid foundations, rather than at the start when you are trying to figure out what the foundations are.

In the software industry the “Discovery Process” is the foundation laying part of any project, get it wrong and all subsequent steps will be endangered.  The project owner has an obligation to ensure sufficiently experienced & empowered resources are made available to the Discovery Process for the required duration to ensure the correct requirements are gathered.  The delivery team has an obligation to ensure the Discovery Process is conducted in a manner that gives the project owner what they need and not what individual users think they want.  There is generally a fundamental difference between what the users say they “want” and what a clients business actually “needs”.

In large Financial Services projects it has become common place to try and get all the requirements at the start of the project and let the users test the result at the end of development.  What users are being asked to do is to define their world now, go away for six-twelve months and then come back and see if the result works as expected …… To be honest I am pretty hazy on what I am doing next month let alone next year and we continually ask users to define their world NOW and deliver it to them in a years time.  Giving people what they consider their ONE CHANCE to define their world leads to mistakes, conflict, wish-lists and myriads of other issues.

We need to stop asking people to try and define their world in one go,  encourage them to move to processes suited to the new application and avoid over engineering a new product to the point what we are removing flexibility rather than creating it.   One of the best means of categorising requirements (but rarely used effectively at the start of the gathering process) is MoSCoW.  We all use some type of categorisation and prioritisation techniques, but this is generally AFTER we have spent 3 months gathering 1,000+ requirements.  This is my suggestion on how requirements should be categorised using MoSCoW before they are gathered to focus the project teams on what to discuss at the appropriate time;

1. (Must Have) What is legally and contractually needed – These are requirements that cannot change as they are enshrined in statutory laws, product definitions, customer contracts, partner contracts & application infrastructure.  To get into this category a requirement must be needed to avoid legal and/or operational issues (i.e. if not delivered the company gets sued or cannot operate).
2. (Should have) What is required because this is how we work now or want to work – These are requirements that you can answer yes to the following question “by having this requirement will the company save money and/or make money and will the saving be more than the cost of development?”
3. (Could have) This is what we want! – These are generally the “Feel Good” requirements and also requirements raised because of  one of the following – “the we are different syndrome”, “this is how we work now and do not want to change mentality” and “everyone else does it this way so we must too defence”.  If not analysed and justified effectively you will end up with a new application that looks like the old one and a cost/manday spend that blows all mandated budgets.
4. (Will not have) The Unknowns – We know something is going to be required but have no definitive details yet.  Never under any circumstances try to guess the future.  Product definitions evolve, new partnership deals change.  Block in time in the project to investigate further but do not try to build for a possible future state.  Change Control mechanisms are used to deal with these areas.  In fixed price contracts these items should be additional to any agreement.  And finally The Cosmetics do not even spend time capturing these requirements, changing how screens look or removing non-used data elements does not deliver value to the project.

The “Must Have” category is the most important and in reality are the ones that should form the minimum-go-live-scope, (i.e. You Must Have These to operate your business) they are only ones that we should initially sit down, document and agree a price/effort on.  Generally the following items fit in here;

• Statutory Requirements
• Product Requirements (Approved by regulator, written into customer contracts)
• Interfaces (External and Internal – but only for critical systems)
• Partner Agreements (Sales Channels, Commission, etc)
• Critical Documentation (Letters, Contracts, etc) relating to operating the business

Get these right and your business should survive without legal ramifications!  Note, anything that is in the power of the company to change, remove and/or defer does not belong in this category.

The second, third and fourth categories are the ones where we currently try to predict the future and end up delivering the past.  Look at a standard long duration project sequence;

Requirements  –>  Design  –>  Build  –>  Vendor Test  –>  Client SIT  –>  Client UAT

There is significant gap of time between the Requirements and the UAT activities.  Everybody tries to get their pet ideas into the system, users try to over design the solution, the 80/20 rule gets thrown in the bin.

In a significant number of projects the UAT phase becomes a “This is what we really wanted” exercise and change control rules the day deadlines & cost go out of control, delivery team and users fight about what is a bug, change, wish, etc ….

What if we were to break the requirements gathering element of a project in two parts;

PART 1 – “Must Have” and critical “Should Have” requirements  are dealt with as a priority.  This is a quantifiable portion of work that can be completed with limited disruption to an organizations day to day activities.  The deliverables from this area are essential to the business.

YOU CAN IN THEORY GO-LIVE AFTER THIS IS COMPLETED, Few people will want to but there is no reason not to deploy the new system.

PART 2 – Requirements that do not fit into the first category are not needed to prevent legal and operational issues.  They are wanted by the users and as such they need to go through a Cost-Benefit-Analysis to determine what is really needed.  Only spend effort when there is a PROVEN NEED (i.e. “the user community must prove their day to day job is harder, longer, more error prone, before the company spends money to develop the stated requirement”). Prove the need then build, not the other way around.

In order to get best value in Part 2 the users should be asked to execute a pre-UAT on the developments from Part 1 and be made to actively demonstrate/prove why a change is required?   However in conducting this exercise a fixed number of mandays should  be allocated to requirement categories – this equates to giving the users a budget in which to work within.  Having a limited manday spend to work with will limit the risk of over-engineering and ending up with a copy of todays legacy application.  By allocating a fixed number of mandays this area can also be done at a fixed cost but the fixed part is in effort not the deliverables.

A number of other considerations for discovery exercises;

• Get a solution architect involved to make sure the correct requirements are developed in the proper part of the Architecture Landscape.  This is essential in situations where you have a highly componentised landscape
• Do not let the requests for requirements be defended by “It is legally required”  make the person show you the legal text where it is stated
• Be prepared to ask WHY and say NO to unrealistic or wasteful requests.  A number of years ago a team I know were forced to build a feature costing 40K USD that generated cents from billed premiums just because everyone else did it that way.  We estimated the client would never recover the cost.  If the client had deferred on the development they would have saved 40K, reduced complexity in the application and been perceived to be looking after customers better than the competition.
• If you do have to say NO to a requirement make sure you understand the users concern and can offer alternatives.
• Raise issues early and stand your ground.  Remember you are the deployment expert not the user.  If you are forced down a route you disagree with get it acknowledged and documented

 

The trouble with Optimism v’s Reality

Many years ago while working as a deployment manager in a Financial Services organisation, I was told the company operates on a principle of Optimism over Reality.  I have never forgotten this comment and sadly it seems to be endemic throughout many organisations.

Setting of expectations at the start of any engagement is vitally important, however on occasions this “Setting of Expectations” is used as a defence mechanism by project sponsors to twist the original agreements to their own favour and ignore the realities of what is discovered as the project progresses.

How many of us have been placed in situations where project sponsors have decided it will take X days to deliver Y at Z cost BEFORE the project has even started and then ignore all findings as they assumed once you started you were committed to their expectation and vision?

The first qualifier we often raise at project commencement is that “we will determine what is possible within the constraints based on requirements, scope, etc.  “.  Everyone involved usually agrees this is a sensible approach and the project begins, however when you produce a report that states it is going to take twice as long and cost significantly more, the first reaction by the people who commissioned the project is one of denial and an insistance that they specifically remember you telling them it was possible to deliver a full system replacement in six months, complete the project in 5,000 mandays, launch a new company in 3 months, build a space shuttle and get it ready to fly in less than a year…… OK, the last one was thrown in just for fun but actually demonstrates the unrealistic expectations we come across on a daily basis.

This situation gets more convoluted in a vendor-client relationship where in some cases the vendor-consultants are faced with defending a rational position against the overly optimistic vision of the client, without having the full support of their own management who are focused on building the relationship at all costs resulting in the continuance of an engagement under unrealistic expectations.

Why does this happen, and more importantly why is this happening on a consistant basis across many organisations? It is becoming increasingly difficult to convince project sponsors and executive management that there are realistic limits to what is achievable and what more importantly what is not.

It should in theory be a simple exercise to get an agreement that if there are single or multiple constraints such as Time, People, Cost then there is a finite deliverable within these constraints. However the excitment, pressure, or prestige of a new project seems to make people agree to very agressive and sometimes impossible goals or to lead to an impression that just because a limit was placed on one component of the project by some miracle whatever is required to succed will fit within these constraints.

As an independant consultant a client once told me that the most frustrating thing was that I rarely told them what they “wanted” to hear, but the reason they engaged me was that I told them what they “needed” to know.  In other words I shattered some of their dreams. This is not something anyone enjoys doing, it is a very difficult position to be in where you face a client and tell them the star project is not going to happen as they planned or that initial optimism conveyed far and wide within the organisation was based on misguided (or unrealistic) initial assumptions.

Everyone hates their pet projects and aspirations being dashed but most people will accept reason if it can be shown that a portion of what they truly need can be delivered in a fast and efficient manner, or that to achieve their full deliverable it is necessary to increase duration, resources and cost.

I believe we need to apply more focus on how we initiate and conduct projects/programs in order to eliminate this misguided approach and make sure all parties involved acknowledge the same set of expectations as early in the engagement as possible;

• Do not commit to what you do not know – no commitments should be made without the right people being present and the right conditions and caveats being agreed

• You are the expert not the sponsor – remember you were probably asked to do the project because of your experience and past successes, you have the authority and more importantly the moral duty to dispel overly optimistic expectations

• Make sure complexity is acknowledged – information needs to be presented carefully and clearly to ensure major initiatives are not perceived as easy, quick, cheap (the over-simplification of projects via powerpoint has a lot to answer for in this respect)

• Define, document and publish the target – ensure proper scoping and caveats are in place and accepted by all parties BEFORE starting, make sure all those involved know what will be included, what will NOT be included and what is expected of everyone

• Deal with issues immediately – if people are not living up to the initial commitments made (e.g. adherence to minimum change & “we can change how we work rather than change the software”) then put a brake on the project and move no further until the commitments are delivered on or the sponsor who made the commitments rescinds them in writing and accepts that the final results will be different than originally agreed. Likewise if you discover something that blows the initial assumption apart, follow the same course of action

• Consistent and clear communication is crucial – delivery organisations and vendors need to be more forceful in ensuring the right message is broadcast up the chain of command. Make sure you communicate effectively what CAN be done and what CANNOT be achieved with the anticipated duration, effort and cost

• The miracle bucket is usually empty – never allow an over-enthusiastic superior or sponsor to over-ride your findings, this is regardless of how far up the chain of command people threaten to take the issues.  Acquiescing to over-optimistic or unrealistic demands with result in YOU being remembered as the person who agreed it was possible to deliver

• Not every deal is worth winning – from a vendor perspective you need to be ready to walk away from deals that are being threatened by unrealistic expectations.  Remember the client selected you, if they are not willing to listen to reason and take account of your experience then you need to weight up the probability  of executing the project in a mature manner – for example; nobody goes to a car dealer, orders a car and then dictates how it will be built or how long it will take to produce

People are generally realistic about what is possible, however human nature and modern business practices force us to get to a winning position where we try to get more for less. As a delivery organisation you know what can be done with a defined duration, resource pool and cost, and you know what it will take to deliver X.  It is your responsibility to ensure the project sponsor and your own management are fully appraised of what can be done and what cannot be done. Failure to do this will not absolve anyone from sitting in the middle of an 18 month project that is 50% completed wondering how to go live next week. By aggressively dealing with the setting of expectations up front and in an open manner you will negate the pain of misaligned expectations

For those that may disagree with the above, try walking in a car showroom with the intention of spending 30K, picking a 50K car and convincing the saleperson to give it to you because you asked to see some cars….. At the moment software deployment in the financial services arena is operating in this mode and projects are suffering due to unrealistic expectations that evolved by the lack of following a proper set of formalised and tough initiation criteria.

A new year, a new blog!

Having been involved in the IT section of the Financial Services industry for over 20 years and experienced both good and bad aspects of system deployment, building greenfield operations, migrations and general consultancy work, I figured it was about time to get my thoughts down in writing. You may not agree with all of what I will write here (or maybe any of it) but I welcome all comments and feedback.

The purpose of this blog is to get discussions started and get my thoughts and approach to system deployment out to a wider audience.

We spend an awful lot of time and money building solutions for problems that are in reality non-critical to the business or focusing on systems when the real solution lies in process and approach.

Its time to stop wasting money and start addressing the right questions, hopefully leading to the delivery of the appropriate answers.

I will not mention names of past (or current) customers nor of colleagues in this blog (unless they wish me to). I would request that any comments also respect this approach.

I look forward to all and any comments on the forthcoming posts.