Software Best Practices

The IEEE is currently requesting papers (sounds so academic) for a Software issues on quality. Their focus is on upfront quality (naming quality attributes, making sure they appear in the product. You can read more about it in a follow-on post. I am sorely tempted to try my hand at writing a paper (though you may think it unwise after reading a few of my blog entries.)

My question to the group is, has anyone worked with a quality model like ISO 9126 and created requirements for quality that do not mention the word defects? That is, a quality requirement is NOT that is passes tests or has less than X defects. Can you name them? How common is it to have identified key non-functional requirements as the quality drivers?

IEEE Software Magazine Special Issue Software Quality Requirements: How to balance competing priorities

Submission Deadline: 1st September 2007 - Publication: Mar/Apr 2008

How do you define quality in your software products?
How do you express the level of quality that stakeholders might desire?
How do you document those desires?
How much time do you spend specifying requirements for quality?
During development, how do you balance the competing demands of schedule, cost, scope, and quality?

To some extent, every software development project must contend with these questions. This special issue is about how software project decision-makers address these questions.

In this proposal, a requirement is defined "an externally observable characteristic of a desired system" [Davis 2006]. While a functional
requirement defines 'what' a system does, a quality requirement defines an attribute reflecting how well a system functions [Gilb 1988, 2005]. Quality requirements include all of the "ility" factors that commonly appear in the literature; for example, reliability and usability. They also include
portability and maintainability; although these are slightly special cases because they relate to the characteristics of the product as a whole, not
just execution time functions.

The elicitation, analysis, and specification of requirements for quality sets the expectation that the software development team will carefully
balance a broad spectrum of competing priorities that often exhibit complex interdependencies and trade-offs. For example, although stakeholders may want a product that executes quickly, the developers must help them determine "how quickly," "within what cost budget," and "within what schedule." The desire for quick execution might conflict with other desirable qualities related to a broad range of qualities such as security, reliability, or usability. Engineering a successful product means balancing competing demands to optimize stakeholders' expectations and  ltimately to increase product value. Balancing the demands infers that the quality attributes be measurable in some way.

This special issue of IEEE Software will investigate issues surrounding the management and trade-off of requirements for quality and the critical role they play for the delivery of a successful software product to the marketplace. Just getting the functional requirements right isn't enough in today's competitive market, where customers want software systems to deliver features with  fast response times, aesthetically pleasing user interfaces, streamlined functionality, and builtin security that doesn't get in the way of the tasks to be performed.

A recent analysis of fifteen publicly available software requirements specifications showed an almost universal lack of any requirements describing product qualities. This suggests that developers and stakeholders may believe requirements for quality to be implicit and therefore may fail to understand the importance of proactively eliciting, modeling, analyzing, and balancing competing qualities.

Stakeholders' needs are frequently not fully articulated and are often left as unspoken assumptions; a situation that can easily result in the delivery of a lackluster system that fails to meet anybody's expectations.

This issue will investigate techniques, processes, and tools for effectively eliciting, modeling, specifying, and analyzing software requirements for
quality to maximize delivered stakeholder value. It will further evaluate assessment techniques for evaluating the quality of a software system in
respect to its stated (and possibly competing) quality goals.

This special issue addresses the following questions:

* To what extent does emphasizing requirements for quality result in the delivery of systems that more effectively meet stakeholder needs, wants, and expectations?
* What effective methods exist for eliciting, modeling, specifying, and optimizing potentially competing stakeholder objectives for quality?
* How can software designs be assessed to show that they will meet product quality objectives?

Submissions may include, but are not limited to, the following topics:

* Techniques for eliciting, quantifying, modeling, analyzing, and specifying requirements for quality.
* Techniques for measuring the degree to which requirements for quality have been addressed in the design and implemented in the final product.
* Techniques for modeling and analyzing the value of design alternatives in relation to a set of requirements for quality.
* Techniques for resolving the conflict among requirements for quality and for balancing, negotiating, prioritizing, and resolving competing demands of quality versus budget, schedule, scope, and other factors.
* Case studies that have quantitatively evaluated the return on investment achieved through focusing on requirements for software quality.

When evaluating submissions for publication, the editors will give significant weight to those papers that report practical and scalable
techniques, tools, and methods. Case studies and actual experience will be considered over theoretical or definition papers.

We specifically avoided the term "non-functional" in this proposal. The special issue will contain an article that argues to abolish using the
almost universally disdained (yet still popularly adopted) term "non-functional" when referring to requirements for quality.


Manuscripts must not exceed 5,400 words including figures and tables, which count for 200 words each. Submissions in excess of these limits may be rejected without refereeing. The articles we deem within the theme's scope will be peer-reviewed and are subject to editing for magazine style, clarity, organization, and space. We reserve the right to edit the title of all submissions.

To submit your paper: Follow the IEEE Software procedures as described in www.computer.org/software/ author.htm#Submission, specifying that you are submitting an article for Software Quality Requirements special issue.
Please also send an electronic copy of your submission to both editors.
For detailed author guidelines: visit www.computer.org/software/author.htm or contact the magazine at software@computer.org.

For more information, contact the Guest Editors:

J. David Blaine
7887 Embry Point
San Diego, CA USA
Phone: 858.349.7691
Email: jblaine@san.rr.com

Jane Cleland-Huang
School of Computer Science, Telecommunications and Information Systems
DePaul University
243 S. Wabash Ave, Chicago, IL 60604, USA
Phone: 312.362.8863
jhuang@cs.depaul.edu

Yes, of course. Most quality requirements are not about defects. There' re only two I can think of: defect density and failure intensity. The quality requirements for which I've had to deal include notions of: Learn-ability, Port-ability, Maintain-ability, Documentation Use-ability; and in telecom: Speech Intelligibility, Voice Quality, Audio Quality (it is different than 'voice' quality).

Hi David, Welcome to the Conversation!

Did you create those requirements ahead of time? That is, did you explicitly state them as goals at the beginning of the project or just experience them as quality requirements somewhere in the project, more interestingly, near the end of the project?

These examples cover many different projects. All of these were created during Project Planning, ahead of time, and were explicity stated as goals. The telecom examples were defined for an entire product line of mobile phones, prior to work on any specific model. Each of the quality requirements had its own scale of measure ("scale" in Planguge) and a set of plan levels that were to be achieved at distinct iterations of the project. All of the telecomm examples achieved, pretty much, their planned levels.

Earl Beede:

Hi David, Welcome to the Conversation!

Did you create those requirements ahead of time? That is, did you explicitly state them as goals at the beginning of the project or just experience them as quality requirements somewhere in the project, more interestingly, near the end of the project?

If I understand correctly, it is best to collect all non-functional requirements (various -bilities, quality requirements are part of them) upfront as much as possible. Functional requirements are less important. Based on the selected non-functional requirements you choose the process, tune it to meet your environment, select tools, estimate budget and schedule.

 
For example, if you have high quality requirements for a life/mission critical software, it affects not only the process you can possibly choose but also development/testing tools, your budget, and schedule. Extra functional features can be added later on if the architecture allows, however, there is no way as far as I know to improve the quality of a desktop application to that of a life critical system.
 

Hi Nick, Welcome to the Conversation!

I fully agree with you. While I think this is right (and what I teach) does anybody do it? Do you do it regularly on your projects? If so, which methods do you use to get them?

Not to draw too fine a distinction, but the emphasis in the Call that was kindly quoted is on the TRADEOFF among quality attributes. I understand that that's not your question, so i just wanted to make sure that the intent of the Call was clear.

Carry on!
 

Earl Beede:

Hi Nick, Welcome to the Conversation!

I fully agree with you. While I think this is right (and what I teach) does anybody do it? Do you do it regularly on your projects? If so, which methods do you use to get them?

Thank you, Earl, it's my pleasure to join a conversation with intelligent people.

> Do you do it regularly on your projects? If so, which methods do you use to get them?

I am graduating with my PhD later this year so I can't brag about many completed projects. I have not found much literature on process selection based on non-functional requirements. There are lots of books and authors that heavily promoting their processes, and it does not take long to discover that their processes are tuned toward one area of software projects and not to another. :(

I will briefly go through the tuning process that I am using right now (brief description). Maybe I should publish it? :)

Given: a toolbox of practices, tools, document templates, etc.
Examples:
a) practices: prototyping, test-first development, agile development, frequent customer feedback, development time measurement/statistics, coding standard, and so on;
b) tools: version control, code analyzer, unit testing software, COCOMO, time counters, UML tools (digital camera, pen, paper) and so on;
c) document templates: IEEE standards, RUP templates, use case templates and so on.

1) Elicit all non-functional requirements as much as possible in the early stage of the project. These should include non-functional requirements from a customer, requirements from management (budget, schedule targets), requirements from developers (leaves due to marriages, divorces, training time).

2) Rank the importance of all requirements. For example, for a small-to-medium size desktop application, usability and schedule target are likely to be more important than testability or very low defect count. For a medical life critical software the priorities may be the opposite. Suppose you picked 20 requirements, assign priority 20 to the first, 19 to the second, and so on.

Important: I avoid assigning the same priority to two similarly important requirements. If I did it would mean that I am postponing making a decision now. I prefer to make a decision right from the start and pay a lot of attention when/if I switch the priorities down the road. Such switching is a direct indicator of either lack of experience or lack of information that I SHOULD HAVE collected from the customer earlier.

3) Based on several top non-functional requirements on your list, select existing processes that have a published history of being used on such projects. Pick the core features of all these processes: agile/plan-driven, iterative/linear, standards, etc.

4) For each practice, tool, or document in your toolbox mentioned at the top you go through the list of your requirements and decide whether it will be beneficial or detrimental to each requirement. Add priority (with + sign) if it is beneficial, subtract priority (with - sign) if it is detrimental to the requirement. Compute the total, if it is positive use this tool/practice, if it is negative avoid using this tool/practice.

For example, your list is this:
1. (Priority: 20) lowest defect count
2. (Priority: 19) shortest schedule possible
3. (Priority: 18) modifiability of the software
4. (Priority: 17) developer training
5. (Priority: 16) budget
...

Your tool from the toolbox is a version control software: "Should I use a version control system"?
1. Version control drastically reduces chances that a wrong source file will be used => lower defect count => +20 score
2. Version control needs deployment and maintenance => takes some time away from development => -19 score
3. Version control does not affect software design => modifiability of the software is not affected => 0 score
4. Training to use a version control takes no more than a few days, most developers are already familiar with CVS but not with Subversion
=> no significant developer training is needed if we pick CVS => 0 score
5. CVS is free and we do not need another person to maintain it => 0 score on budget
...
---------------------------------------------------------
Total: +1 => YES, we will use CVS version control on the project

Many "best practices" are "best" because they have little to none detrimental effect on non-functional requirements. That is, you will automatically end up using best practices on most projects.

5) When you are done, you will have a core process with a collection of documents, tools and practices that you consciously selected to bring value to your project, instead of selecting those that a process forces upon you.

> While I think this is right (and what I teach) does anybody do it?

If you are looking for a person with extensive hands-on experience in the role of a project manager ... I don't think there are many people you can find.

For a person to be able to make such a decision based on his previous practical experience, he should have worked as a project manager on various projects ranging from simple web-based plug-ins all the way upto mission critical projects like shuttle control software, and without getting promoted to a higher managerial position. That is, he should be a life-time project manager. This is highly unlikely. On top of it, the industry is divided into areas where some or many of non-functional requirements and corresponding processes are taken for granted, e.g. aerospace industry had CMM level 5 history/certification (minimal defect count), web projects used XP ideodology (flexibility, usability), systems software projects used RUP (large-scale fixed architecture, flexible on features, schedule predictability), in-car software projects may have used Scrum (small highly-specialized softwares, flexible) etc. Any project manager will have a hard time switching from one industry to another. Can you imagine a web project manager with XP history to be hired to do a medical project (life-critical software)? The easiest way to find people with wide experience is to look at software consulting companies that are not biased towards a specific process (if there are such companies). Steve McConnell's books have this nice feature of being process-neutral, which is probably why they rank so high on Amazon.

In my opinion, software engineering concepts that you are teaching help first-time project managers make a minimal number of mistakes. For young people like me, coming into the industry with the basic knowledge of 'why', this is a huge asset. It is even a bigger asset for those who start up they companies now right after the college. This knowledge may give them a competitive edge they need to drive their competition out of business.

Hello Stan, Welcome to the Conversation!

You are right, my question is slightly different than the Call. I figured that before I put a tremendous amount of effort in trying to write an article, I would check to see if doing non-functionals early is one of those things that, while correct and a good idea, just isn't done in the industry. If you look at most of the practice defintions (except Gilb's evo), there is almost no mention of going after non-functional requirements. The best I can see in the current popular Agile methods is that some people suggest that you ask about "conditions of satisfaction". "Stories" themselves are focused primarily on functional items.

My approach to an article so far is the things I have learned trying to teach the approach of focusing on non-functional attributes to groups. Most folks are really surprised at just the idea.

Hi Nick,

Your process sounds close to Gilb's evo process. He scores designs instead of (or in addition to) practices but I bet he would really like your approach as well. You should share your stuff with him at tom@gilb.com.

I had not thought deeply about how the methodology itself emphasizes some product non-functionals. I think projects have requirements as well and so I just associated the methodology with project requirements. I will need to think on methodology non-functionals impact the product some more.

All,

You may have noticed one short paragraph at the end of the call: using the term "non-functional." Although there is some precedent from many years ago when people didn't know any better, the adjective "non-functional" is abhorrently inappropriate. Nonfunctional is an actual word in the dictionary with an actual definition. It means "it doesn't work."

Using this term in the context of requirements is undesireable at the least. Lumping all requirements that are not about 'functions' into one witches' cauldron of "non-functional" is not responsible. Depending upon the particular author, standard, or viewpoint there are many categories of requirements that are not about functions: Programmatic (or "process") and Resources are two examples. Similarly, two examples from Alan Davis are requirements about "Objects" and "States." Are these "nonfunctionals"?

In most of the articles I've read, and most of the blog type threads, the context clearly shows that the topic is Quality (or "Performance," the term which systems engineers like to use) requirements. There is great value in being specific and very little value in mis-using a word that literally means "not working." The level of discourse will be improved if you say what you mean.

Do you really want to brag to a customer that your product satisfies "nonfunctional" requirements? Would you buy a car from a salesman that boasted "Yes, this car has complete nonfunctional coverage"?

Earl Beede:

Hi Nick,

Your process sounds close to Gilb's evo process. He scores designs instead of (or in addition to) practices but I bet he would really like your approach as well. You should share your stuff with him at tom@gilb.com.

I had not thought deeply about how the methodology itself emphasizes some product non-functionals. I think projects have requirements as well and so I just associated the methodology with project requirements. I will need to think on methodology non-functionals impact the product some more.

 To Earl,

Thank you for pointing out similarities with Gilb's evo process. I will
definitely ask his opinion on that.

I am not sure that I completely understood your message though. I do not
have any process and I am not building one although it seems like a new
trend now in construction of software development processes. Perhaps, I
did not make it clear but my procedure may lead to a planned process as
well as an iterative one. Thus, unlike Evo it is not a process, but rather
a procedure to construct a process on-the-go for each specific project,
it is not always iterative, and it does not set any predefined iteration
lengths. It DOES lead almost always though to a measurement-heavy process
like Evo or PSP/TSP processes. While it may be perceived as a micro-management,
I believe that measurement is too essential for proper schedules, quality
control, and rush-free development to easily abandon it. As long as this
measurement is not used for employee performance evaluation it is OK.

To Dave,

It is important to use proper words, however, every industry is ripe with
inadequate adjectives. The question is whether it is worth an effort to
change these words? Has it ever caused any problems in customer liaison?
If not then it is likely not being worth the effort.

In addition, if you think logically none of these sentences make sense:

"Requirements work" or "Requirements do not work"

A requirement is a non-operating entity as such it can exist, but it can't
work. Therefore, when we logically use phrase "non-functional requirements"
we must be implying something other than "work" or "do not work".

Second, neither "quality", nor "performance", nor "process" include all
"non-functional requirements". From a logic perspective, the set of
non-functional requirements is a complement of the set of functional
requirements and as such it is well suited to include all other requirements.
:) I hope I have successfully persuaded you that black is white and white
is black?

Yes, it has caused problems with customers. And, no, the term 'non-functional' is not a useful compliment to requirements about functions. As stated, requirements about 'objects' are in the set of all requirements that are not about functions, but to refer to them as "nonfunctional" is silly and does not add value to readers or developers.

Referring to quality attributes as "nonfunctional" is not helpful to anyone. To lump all requirements that are not about functions into one subset is to demonstrate a lack of understanding of the role of requirements and is not conducive to engineering.

The adjective "functional" does not precisely modify the word requirement. Functional requirements could be more correctly called "Function Requirements" (as Gilb does) because they refer to the functions of the system.

Pleading that inadequacies exist in other disciplines justifies continuing a bad practice is not a valid argument. It does not take much effort to abandon the ridiculously inept term "nonfunctional" when, in fact, the writer is referring to "quality" attributes. If the topic were objects, then "Object Requirements" would be the appropriate term. It is precisely because the set of all requirements that are not 'functional' requirements is so diverse that more specific adjectives are needed.

I hope I have convinced you to adopt a more precise discourse to help raise software development towards the level of engineering.

While I'll agree that Non-Functional is not a good term. I can't that I am very excited by Quality either. I have tried using it and people get a confused look on their faces. "Does he mean defect count?"

Then, to break into 35 sub-groups, that doesn't help much either. "Program, Program, can't tell your requirement without a program."

For what it is worth, I start in my requirements seminar as the Requirements That Are Not The Function Ones, Often Far More Important.

However, it still doesn't seem that anyone is actually focused on them at the start and working them through the project.

This all seems a bit like much ado about nothing when we have much bigger challenges in the drive for software quality.  However, I think of the various terms for requirements as being a short hand for somewhat lugibrious phrases like "function-oriented requirements", "performance-oriented requirements", "security-oriented requirements" and "not function-oriented".  Thus I think the terms "function requirements", "preformance requirements", "security requirements" and "not function requirements", respectively, make sense to me.  Still, most people are quite comfortable with the terms "functional requirements" and "non-functional requirements", and I have never encountered a situation where this was the cause of confusion.

 On the other hand I have encountered situations in which a poor taxonomy of requirements caused confusion and much worse.  For this reason it seems counterproductive to divide requirements into "function requirements" and "quality requirements".  While there are even narrower interpretations of "quality requirements", in actual fact all requirements contribute to quality.  The phrase "quality requirements" is redundant.

 I usually categorize requirements along a number of dimensions, one of which is "type".  At the high level, I usually use function(al), interface, reliability, security, performance, usability, testability, maintainability and portability (as side note, I have usually found that design for testability - when done correctly - results in maintainability and portability).

I apply this to testing as well.  Specifically, I create a taxonomy for testing which includes the type of requirement being tested (as per the above), the level in the system hierarchy of the item under test (for example, unit, component/assembly, subsystem, system and end-to-end system) and the operating range of the item under test (for example, nominal and stress).  By applying this taxonomy heuristically, I can ensure that I identify all possible kinds of tests.  This helps me make rational decisions about where to allocate test resources and time, what types of test tools and harnesses are required and what types of expertise are required.

You have missed my point. My objection was dividing the universal set of requirements into just two sets: "functional" and the silly term "non-functional." 

There are many categories of requirements that are not 'about functions'. I gave several examples. I asserted that in many (I think most) cases where the term "nonfunctional" is used, the actual subject was "quality" or "performance." I have not seen many instances where "nonfunctional" was intended to include "Programmatic" requirements for example.

Quality Requirements are probably more rightly termed Quality Attributes because they modify functions. Functions state "what", quality (or performance) attributes state "how well." The "how well" category includes all of the -ility adjectives, including those that constrain the source code, like 'portability'. But there are, of course, other categories of requirements for example "Programmatic" or "Process" which are clearly not quality attributes.

Gilb, in Competitive Engineering, offers fairly simple and reasonable categories. So, for example, does Alan Davis.

The distinction between functional requirements and quality attributes is clear and important.This may seem to be 'much ado' only because some writers have not experienced face to face meetings with highly motivated project sponsors with millions of dollars at stake in ultra-fast moving and competive environment. Standing up in front of such a group and crowing about "nonfunctional" is a non-starter and a quick terminator.

Dave,

I think that is right. We have functions and we have quality attributes of those functions. Often on a project, the "function" remains the same and only the quality attributes change. One client I have calls them System Characteristics.

While I am tempted to move this whole thread to the requirements forum :-), is anybody seriously practicing a way to identify those attributes/characteristics early in the project and focusing the project on delivering those characteristics?

dblaine:

You have missed my point. My objection was dividing the universal set of requirements into just two sets: "functional" and the silly term "non-functional." 

There are many categories of requirements that are not 'about functions'. I gave several examples. I asserted that in many (I think mos