Agile Curam Development Requirements Gathering: Activity Scenarios

In my previous post I talked about end-to-end scenarios.  In this post, I am going to talk about what Activity Scenarios are and why they are important.

An Activity Scenario expand on one or more workflow activities from the end-to-end scenario.

By putting this into a separate task, we make sure that we help ensure that we do not get bogged down in the details early on.

Input sources for Activity scenarios

The main input to the activity scenario are the end-to-end scenario which shows relatively how things will fit into the entire scheme of things.

In addition, the following inputs will help expand on them.  This is relatively the same as the end-to-end scenario inputs, because these would apply throughout the entire requirements gathering process.

• RFP from the customer
• User Stories written by the customer for each activity scenario.
• Previous output products if any.

Output work products for ACTIVITY scenarios

The output for each activity scenario is a detailed description of what the activity involves.  Generally there are three major parts for each activity scenario.

• Manual Activity information
• Automatic Activity information
• Control Activity and linkage information

As I specified earlier there may be more than one workflow activity from the end-to-end scenario involved in an activity scenario.  Allowing for it would allow people to work things in a slightly higher level than individual activities, but still lower than the end-to-end activity.

The output is to be given to the developers to implement the workflow.

Planning

The Lead Pair would define the activity scenarios and send them to the PM for the planning.  The pair would to decide whether group more than one activity in an activity scenario.

The client with the PM prioritizes activity scenarios that they would like to elaborate on.  Generally there should be little interdependencies between activity scenarios so these can be done in parallel.

OUTPUT: AUTOMATIC Activity information

An automatic activity user story is simpler than a manual and it just describes what the system should be doing based on information that it has.

Output: Manual Activity Information

A manual activity is a bit more complicated as it involves determine who should get the task, the deadlines, etc.  What a BA needs to do is break down the user story into the components of manual activity.  Although you would be tempted to make use of a template that the customer fills out, I would make sure that parts of it are free form text where the customer can write a story as to what should happen when determining the deadlines, who to assign it to, etc.

When defining your first few manual activities (maybe the first 5) try to see if you can create a common manual task pattern that can be extended rather than repeating yourself for the rest of the project.

Having a common manual task pattern would also allow the leads to say:

Anything that we haven't explicitly defined or built we will use the common manual task pattern.

This would allow the requirement gathering team some air by:

• allowing the developers to work on the workflow that can be tested.
• allowing customer/test team to test the flow, even if it is not fully automated yet.

Output: Control Activity and linkage information

As specified earlier, an activity scenario may involve more than one activity.  The information obtained here is used primarily in defining loop conditions, event transitions, etc.

Conclusion

As specified earlier, Activity Scenarios expand on one or more workflow activities from the end-to-end scenario.  This task separation prevents analysis paralysis.

Since each activity scenario is usually a stand alone unit, it can be prioritized based on importance.  In combination with the common manual task pattern, any activities that are unable to be met by the deadline can be postponed.  Since the workflow is still present, it just means that some people need to do some work rather than having the system do everything.  However, since you can deploy new workflows during the life of the application, there is little risk.

Although this is primarily for Curam, most workflow packages would be able to support the activities specified below as it is technology agnostic.

Agile Curam Development Requirements Gathering: End to end scenarios

This is a continuation of my previous post in which I specified how I would break up the business analysts with the customer.  The approach on getting the requirements out uses the same process as User Stories in Extreme Programming.  The following is a recap of the user story groups that would be done in Curam:

• End to end scenarios
• Activity scenarios
• Product/Rules/Evidence (for non-Curam projects this may not be necessary, but other projects may need to define the data and the rules behind the data from a domain expert).
• Usage scenarios

In this post, I would be talking about the first group which is the end to end scenarios.

An end to end scenario is representative of a full workflow as told by the customer.  It is best to get an idea what the end to end scenarios are first, without it there isn't going to be a good understanding of how things would flow through the application.  Without that, you'd just end up with chaos in development.

Having end to end scenarios defined makes things a lot easier to develop and test.

Input sources for end to end scenarios

There are three possible sources for end to end scenarios:

• RFP from the customer
• User Stories written by the customer
• Previous output products if any.

Output work products for end to end scenarios

There are two possible work products from the end to end scenarios:

• High-level workflow.
• Sub-workflow requirements.

This gathering information for the end to end scenarios is an iterative process, the output work products feed back into the process as new input.

High-Level workflow

A high-level workflow would at minimum capture the following:

• How does it get started?
• What information do we have when we enact the workflow?
• What are the activities?
• Who are assigned to each activity?
• Are there any activities that can be split into a sub workflow either because it is vague or to complex?

The workflow is then created and visualized using the Curam workflow activities.  The workflow can then be shown to the client to confirm the understanding of the scenario.  This task needs the BA to be cognisant of what the Curam workflow can provide.  The workflow would also determine who are involved.

Although it would be nice, there are some information are not immediately required to complete a high-level workflow.  It is best that the these extra information would be detailed by a different group of BAs.  Otherwise you'd get into analysis paralysis of the end to end workflow and won't get anything done.  They are:

• What kind of information would we need per activity?
• Notification requirements.
• Distinct names for each element of the workflow.

In the en, the work product would be a document that would have the following sections:

• Workflow diagram
• A section for each activity
• Enactment information

Once the document has been signed off, it is sent down the pipe to the activity bucket.  Which will detail off each activity on hopefully a different document, but tied to the original work product.  The reason why you'd want it as a different document is to ensure tracibility.  That is:

• If the customer decides to change the original work product, we know that the activity document would need to be revisited.
• Accountability, we want to ensure that the detailed version keeps the original information intact.

The activity document should have the same content as the original one though, it would just have new detailed information added underneath it.

Sub-workflow requirements

Determine sub-workflow requirements should not be too difficult.  One of best ways of doing this is to define a metric.  A good metric would be that the workflow diagram must fit within one legal sized paper in landscape orientation.  By limiting the size of the workflow, you'd be forced to write sub-flows in order to handle complexities.  The advantage of this would be better focus for the customer and the BAs as they won't get too bogged down in the details.

Imagine having workflow diagrams the size of the Curam reference model poster, you'd probably have difficulty following it.  Even worse if you are a new person to the project.

Working with the customer

Many of these scenarios be done at the same time, but the customer needs to prioritize them.  It takes probably a couple of days to draw a 50 activity workflow by someone that has done it once before.

The cheapest ways of building the workflow are to do it on a shared piece of paper or transparency sheet for larger audiences.  Then have someone draw it more professionally afterwards using Curam.

When capturing the detail for an end to end scenario, make sure the customer's focus is going from end to end, rather than expanding every single detail which may take a longer time.  Explain to them that the parts they are not 100% sure of or would require a lot of steps can be broken into a sub-scenario (which would be coded as a sub-workflow eventually).  Doing this would avoid analysis paralysis.

Non-Curam projects

For non-Curam projects, try and convince your customer and architect to invest time and effort on implementing a workflow solution.  There are several out there, some of them free.  Others like WebSphere Business Integration/Modeller provide a lot of overkill, but it is good overkill.  Just like DB2, it has an overkill of features, but that is what makes it strong and able to deal with many of the things you throw at it.

However, even if you do not have a workflow system, having end to end scenarios defined makes things a lot easier to develop and test.

Conclusion

The output work products for the end to end scenarios will provide a framework for you test team to follow when they are actually testing the application.

These scenarios will help foster and reinforce understanding of the customer's business both to the BA and the customer.  Visualization is a very powerful learning tool.

By breaking things into activities, the other BAs can be more narrowly focused, which means that the development team would also be narrowly focused because the flow would be handled by the workflow engine.  Also since each activity is more well defined, it is easier for the PM to plan and set dependencies between each activity.

So as you can see having end to end scenarios defined makes things a lot easier.  However, not just to develop and test, but planning and fostering customer understanding as well.

Agile Curam Development Requirements Gathering: Working With Your Customer

This is a continuation of my previous post.  In this entry I talk about working with your customer in hopefully an effective fashion.

One of the toughest part in any project, not just Curam, is getting enough information from your client so that the development team can use it and the client won't come back and say, "that's not what I meant."  This may seem like squeezing blood from a stone, but when done right it won't be as bad.

The thing that should help on the get-go is planning and defining who does what.  The other important thing a Curam BA has to do is to help your customer adapt to Curam, rather than accepting everything they want.

Unlike non-Curam projects, this is a lot easier for Curam practitioners as they already have a working product that they can use and play with.  However, regardless of whether it is Curam or not, it is best that the BA works with the application architect to make sure decisions that they make won't have major consequences to the work.

Roles and responsibilities

It is best to determine roles and responsibilities as early on as possible.  Customer business reps should be talking with each other, same with your BAs.  However, one of each side must lead their own packs.

Here are the pair levels:

• Lead pair - talks about the end to end scenarios with each other.  Creates the first workflow drafts.
• Workflow pairs - talks about workflow and activities to a lower level of detail.  Joins with the lead role meetings to get an overall understanding and confirmation on what the scenarios are supposed to be doing.  You may end up with one pair for every 1 or 2 end to end scenarios.
• Delivery pairs - talks about the deliveries that would need to be done and the rules and evidence associated with them.  You may end up with one pair for every 1 or 2 deliveries.
• Usability pair - talks about how things should flow screen wise.  They would also talk about modifications to existing screens to correspond to privacy guidelines.  They would also talk about who would have access to what screens.  This is usually just one pair but works with the other pair levels to understand how the screens would relate to the business.  There may be more than one pair if the customer wants to have a lot of changes done to the screens and security.

If the project is small, which is rare for Curam, some roles may be combined into one person.  On the other hand, if you have a big enough project, it may require more than one person to do the role.

Planning

Now that you know who your resources are, you should be able to create a project plan with dependencies on each component.  This is necessary in case someone misses a deadline.  If a deadline has not been met, then the project plan needs to be re-adjusted as a change request.  Missing a deadline is a change in scope and should be treated accordingly, otherwise you'd end up with scope creep.

I would recommend that you use a proper planning tool such as Microsoft Project or OpenProj (when it becomes less buggy) to help plan things out.

Also you need only to take the detailed plan on the requirements gathering portion.  It is not too bad unlike other areas, because you really have no external dependencies short of the customer.

90/10 not 80/20 rule

You do not have to push the client to finish things before they are done.  Remind them that if they spend too much time trying to get things perfect at the beginning, then they'd cause more people to be idling.  However, they should not leave things to be in an imperfect shape, otherwise they'd still have to pay for any changes to their original requirement once development has started.  It is a balancing act.

Fortunately, the approach I am taking uses the User Story method that they do in XP for most of the work products a BA needs to build.  This approach gets them to say in their own words what they want to accomplish.  These may have already been put into the RFP, but even if it is not then we would have marked the component as high complexity and so more time can be allocated to get the necessary information.  The other nice part about User Story method is the BAs don't have to write long wads of documentation, that task is passed onto the customer, who should know how they want things to work more than the BAs.

Because Curam has done a lot of things for us already, a 90/10 target should be strived for rather than 80/20.  Of course the best way of doing this is...

Adapting to curam

Keep in mind, just because the customer wants something, does not mean that it should be how it is supposed to be done.  Especially with the usability and case lifecycles.

Using a COTS package like Curam implies that there must be some flexibility with the customer to adapt to Curam.  Otherwise, Curam should not have been proposed in the first place and instead a custom solution should be built.

Meetings

The best layout with regards to meetings with the customer and the BA would be as follows

• Monday - All hands meeting, to plan things to do during the week.  BA team then has a meeting with other groups in the project, specifically development.
• Tuesday - Meet up with the customer business rep.
• Wednesday - Meet up with the BA team to make sure everything works.
• Thursday - Meet up with the customer business rep.
• Friday - Meet up with the BA team to make sure everything works.

The reason why it is laid out in that fashion is to give the BAs enough time to prepare for their next meeting with the customer.  Consecutive days of gathering customer requirements would not be the best use of their time because there would be too much things to capture and not enough time to think about what they captured.

Conclusion

A colleague at work was also suggesting I try to write up this series as what "I would like for a BA to give me as a developer" rather than being Curam specific, so I have tried to do above by putting in things that are non-Curam specific.

The following points are not Curam specific:

• With proper roles and responsibilities defined, the risk of miscommunication is greatly reduced.
• With proper planning, scope creep caused by slow customer response is reduced.
• It does not need to be 100% perfect, but don't slack on it either.
• Knowing the technology would help you make better requirements.
• Planning how to meet will help prepare your time for weeks ahead.

In previous projects, I've worked with BAs that give me 99.9% complete and correct specs and I am amazed by the quality.  So at least I had some things to go by. 

It is good that they were helpful about it too, if there was something I didn't understand I just came up to them and they pointed me to the right place to look.

Although I do remember one their comments when I do find something wrong: "We don't like Archie as he reminds us that we are not perfect." :-)

However, reality is what development will actually bring out.  So if I were a BA, I would have to bite the bullet and tell the client about the error.

Agile Curam Development Requirements Gathering

Gathering requirements in Curam does not appear to be an easy thing to do.  One can go with a BDUF (big design up front) approach or an iterative development approach.  With a COTS package like Curam a combination of the two approaches would be more beneficial.

I've seen a lot of BDUF (big design up front) type documentation in a lot of my projects.  Unless you are extremely good at estimating and your customers are very accomodating, these documents tend to take too long to create and is not 80% perfect when they reach their original deadlines.

Iterative development such as RUP/XP try to remedy that.  However, unless your sales team is good enough to sell a time and materials contract, then an iterative development approach from requirements gathering to actual development would be a hard sell as well.

Unlike the traditional development approach where you would almost start with scratch, a COTS solution offers a lot of things for you out of the box.  However, you would need someone who knows what you can get with minimal changes out of the box.  In such cases, you would need more people doing business analysis rather than actual developers.  The BA vs Dev mix should be roughly equal, with some of the BAs transitioning to a test team or Curam administrative roles as time progresses.

I am writing this, in case I do switch to a BA lead role with focus on requirements gathering and analysis in a Curam project that may be fixed price.

I say lead, because Curam requirements gathering for any reasonable sized application is not a single or two or three BA job, doing so is asking for a lot of trouble.  Conversely, it is not a single or two or three customer representative job, doing so is asking for a lot of trouble as well.  It should also be one BA to one customer business rep.

Although my experience in COTS is primarily Curam, I think the concept would apply to most other COTS.

This is the first in a series of blogs regarding requirements gathering.  I thought it best to start at the point before you even start working...

Proposal and estimating

Estimating the requirements gathering from an RFP depends highly on the quality of the RFP itself.  The following list shows a grid of how I would capture things.  The numbers represent some factors I would use in terms of days when calculating things.

	High Complexity	Medium Complexity	Low Complexity
End to end scenarios	30	20	10
Activity scenarios	15	10	5
Delivery definitions	10	5	3
Rules/Evidence	30	20	10
Usage scenarios	15	10	5

Based on the data, I apply the factors to the counts I would gather from the RFP and just sum things up + add whatever buffer factor I would need.

Complexity is determined by how complete the RFP is in the first place.  If there is sufficient detail to generate work products for each component, then I would mark them as low complexity, unless it is really complicated.  Remember, I am not building the final application, but rather gathering enough information so the development teams can do their work.

Please note that what I have written above talks about effort.  It does not talk about dependencies so things may shift further in duration because of such dependencies.

Programming vs Scripting

Normally I would agree with Larry's comments, but after working a few years in both scripting (Perl, Ruby) and programming (Java, C, C++) in many different language variations.  I think if I had to choose between one or the other, it would be it depends.

If I had a choice, it basically boils down to who is going to code and maintain it.  If the answer is anyone but me, myself and I, then I would go for a programming language rather than a scripting language.

Easy Onramps

Though I have to admit, the amount of effort to learn Perl myself is quite shallow, I learned to do most of the things I need to do within the first week of using Perl just because I code-build-run cycle is just one command, perl.

When I have to deal with other people, they would already know how to do the javac-java or at least use the Run As Java Application command on their IDE.  Without those, I don't think we'd even get them into our development teams.

Java/C# programmers are generally a dime a dozen.  Though you do get what you pay for.

So providing an easy onramp is good and all, but it is not a reason for me to get other people who are already trained as Java programmers to switch over.

TIMTOWTDI

"There is always more than one way to do it," is a very useful feature when I am coding for myself.  Not just coding, but I think well developed UIs should also follow that approach which I alluded to when I talked about Input Context Efficiency.

Having the option to do things differently reduces the amount of time I have to think about to get something working and focus more on getting what I need done.

When you are working in a large enough team, that freedom can backfire onto you.  Unless you are significantly diligent in your code reviews, TIMTOWDI can create a maintenance nightmare because everyone codes things their own way.

In Java, we have one conditional keyword: if.  In Perl, we have if and unless.

In Java, we have one way of writing "and": &&.  In Perl, we have && and and.

In Java, we have only one way of structuring our code: package containing classes containing methods.  In Perl, we can have an entire method in one file, or multiple classes of different names in the same file, multiple modules in a single file.

In Java, we have to explicitly define variables that we use.  In Perl, we have $_.

Granted that most of these problems can be handled automatically by some code review tools that are available in Perl and Java, I haven't found one for Perl that would integrate with the IDE as Checkclipse does.

early binding / late binding

Most scripting languages have a concept of late binding.  Which means that unlike Java, classes are not "final" and you can add new methods dynamically.

Although if I write code I may need to use that feature.  I would delay finding out about any errors until run-time.

If it were just me, then I would be creating some unit tests that would prove that the methods I call do work correctly.  Or sometimes since I would write something one off, I just let it fail at run-time when I run a test because it is easy enough to find the problem.

However, keeping track of 1000 methods in your head can be quite daunting too.  Not to mention typing it correctly.  Copy and paste may be safer, but it forces me to switch to my mouse to point to what I need to copy and paste.

Auto-completes do help out, but if one just thinks he typed something in correctly, and it fails at runtime.  Then the repercussions could be quite bad if it had been deployed to a production environment where change management processes are enforced.

I'd rather have something fail as early as possible rather than later.  Unit testing does help out with this one, but finding those kinds of developers are more difficult.  Especially if you have to deal with global resourcing.

Conclusion

As I have stated before, it basically boils down to is it for myself or for others.

If I have to write for others, I would use a programming language that would limit what I can do, and have almost everything defined including how to indent and casing.  Because that is easier for me than explaining how I do something in a language most people won't understand.

If I have to write for myself, I'd use whatever tool makes it easier for me.  Which in 90% of the time is Perl.

Logging in Curam

Curam uses log4j for logging.  By default it only logs stuff that you put in using curam.util.resources.Trace to SystemOut.log.  However, since it is using log4j for logging you can tweak it in many different ways by putting your own log4j configuration file.

Setting up log4j configuration

There is a property that you can change using the web UI or through Application.prx called curam.trace.configfile.location.  This specifies the path to a log4j xml configuration file.  The following is an example I got from the log4j wiki that you can use to get yourselves started which will dump practically the same data as the standard logger.

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
  <appender name="console" class="org.apache.log4j.ConsoleAppender"> 
    <param name="Target" value="System.out"/> 
    <layout class="org.apache.log4j.PatternLayout"> 
      <param name="ConversionPattern" value="%-5p %c{1} - %m%n"/> 
    </layout> 
  </appender> 

  <root> 
    <priority value ="debug" /> 
    <appender-ref ref="console" /> 
  </root>
  
</log4j:configuration>

Tuning log4j configuration

Of course you'd probably want to tune things in logging like:

• Only show level of "ERROR" and above for Root logger
• Only show level of "WARN" and above for "Trace" logger.  The "Trace" logger is where Curam sends its trace logs when you do Trace.kTopLevelLogger.
• Only show level of "ERROR" and above for "curam.customapp" logger.  This is an example of a custom application logger.
• Show the timestamp.

This ends up being:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
  <appender name="console" class="org.apache.log4j.ConsoleAppender"> 
    <param name="Target" value="System.out"/> 
    <layout class="org.apache.log4j.PatternLayout"> 
      <param name="ConversionPattern" value="%d %-5p %c{1} - %m%n"/> 
    </layout> 
  </appender> 

  <logger name="Trace"> 
    <level value ="warn" /> 
    <appender-ref ref="console" /> 
  </logger>
  
  <logger name="curam.customapp"> 
    <level value ="error" /> 
    <appender-ref ref="console" /> 
  </logger>
  
  <root> 
    <priority value="error" /> 
    <appender-ref ref="console" /> 
  </root>

</log4j:configuration>

Just don't forget that trace_ultra_verbose messages will not appear with the above configuration so you'd lose some logging.  So how do we get around that?  Well thanks to log4j, you can create a separate appender to put stuff in there, but since Curam does not include log4j-extras.jar so org.apache.log4j.varia.LevelRangeFilter is not available.

Custom application logging

There are two ways of doing application logging in Curam.

• The best way to set up logging in for your custom application is to create a custom application logger so you can dictate the structure of what gets logged.  However, you need to do some extra coding.
• The easy way is to use Trace.kTopLevelLogger so you don't need to set anything else up.  But you lose the flexibility that log4j provides.

The choice generally depends on the logging "control" requirements of the solution.

To perform custom logging, all you have to do is standard log4j.  That means

// Define a logger
private final Logger log = Logger.getLogger(this.class);

// Use the logger
log.trace("hello world");

I just define the logger within a private final variable in the class.  We don't need it in static usually because most Curam process classes are not meant to be executed in a static fashion.  Plus that line can be copied and pasted without having to change anything.  If it were static then we need to explicitly specify the class name.