I spent ages recently googling for references to 'Diabolical Problems', about which I thought I had read. Thanks to Matt Heusser's latest post, I now know I should have instead been googling for Wicked Problems. Just in case I've sent anyone off on the same wild goose chase, this post should set them straight. For everyone else, enjoy the reading.
03/10: Claims testing in the wild
As yet another poor internet soul is scammed by a man pretending to be a woman in online chat rooms, I'm reminded of the sensibility of my number one internet heuristic.
Jared's first law of online safety is 'Assume that everyone you are talking to online is a man'.
This has held me in good stead over time, but how relevant is this to testing? Well, it's strongly related to the testing technique of claims testing. Claims testing is what you're doing when you are testing a product to specifications or requirements. It is perhaps the most common test approach in large corporate environments I encounter, but it's also a technique that can be applied in a shallow, context-free way.
James Bach has two points in his Heuristic Test Strategy Model that I find key in describing claims testing -
- Verify that each claim about the product is true.
- If you’re testing from an explicit specification, expect it and the product to be brought into alignment.
Perhaps closer to what we are actually doing is that we usually verify that each claim about the product *can* be true, for some particular situation or situations. We also try to understand for which cases and contexts the claim holds true. And in order to verify the claim, we set about collecting *sufficient* evidence of the truth of the claim, because exhaustive proof is either prohibitively expensive or impossible.
An example I like to use to teach how claims testing should work begins with me making the claim 'My name is Jared'. I follow with the question 'How would you set about finding out whether it really is true that my name is Jared?'
The example is not as straightforward as it may seem, because the level of evidence required depends entirely on the purpose for which we need to know.
If I've introduced myself online in a chatroom or in person, and you don't care about any interaction beyond that room, my assertion that my name is Jared may be sufficient.
If you're the government of Australia, then your standards are a little higher depending on the service you're providing to me. Sufficient evidence may include my birth certificate or passport, an assortment of historical information, and other knowledge of the details that the government has stored about 'Jared'.
In other instances, social proof may be OK. The fact that others refer to me as 'Jared' may be sufficient evidence, and we can build up a body of evidence based on a history of social interactions.
Or perhaps you're a hitman paranoid about bumping off the wrong person, so you brute-force the solution. You stalk me for a month and rummage through my mailbox, finally whacking me on the head and rifling through the contents of my house and wallet.
Each one of these approaches may be required for sufficiency, and is appropriate for different contexts and purposes. And we might take similar approaches when we're testing software. We might simply talk to people who can provide evidence of a claim being met. We might perform a simple confirmatory test for a non-critical function. We might spend a longer time collecting a large body evidence to support a claim being met. It's important that we think about the importance of the claim and ensure that our approaches to gathering information can be defended under scrutiny.
The real-world analogy to the second point about claims testing - 'Expect the spec and the product to be brought into alignment' is perhaps more commonly found in legal circles, and occasionally in the forum posts or blog rants of less careful men who feel their masculinity has been threatened. It tends to involve situations like this. And as in a situation detailed here that more closely parallels software testing, remember that refuting a claim can sometimes have much bigger consequences.
Jared's first law of online safety is 'Assume that everyone you are talking to online is a man'.
This has held me in good stead over time, but how relevant is this to testing? Well, it's strongly related to the testing technique of claims testing. Claims testing is what you're doing when you are testing a product to specifications or requirements. It is perhaps the most common test approach in large corporate environments I encounter, but it's also a technique that can be applied in a shallow, context-free way.
James Bach has two points in his Heuristic Test Strategy Model that I find key in describing claims testing -
- Verify that each claim about the product is true.
- If you’re testing from an explicit specification, expect it and the product to be brought into alignment.
Perhaps closer to what we are actually doing is that we usually verify that each claim about the product *can* be true, for some particular situation or situations. We also try to understand for which cases and contexts the claim holds true. And in order to verify the claim, we set about collecting *sufficient* evidence of the truth of the claim, because exhaustive proof is either prohibitively expensive or impossible.
An example I like to use to teach how claims testing should work begins with me making the claim 'My name is Jared'. I follow with the question 'How would you set about finding out whether it really is true that my name is Jared?'
The example is not as straightforward as it may seem, because the level of evidence required depends entirely on the purpose for which we need to know.
If I've introduced myself online in a chatroom or in person, and you don't care about any interaction beyond that room, my assertion that my name is Jared may be sufficient.
If you're the government of Australia, then your standards are a little higher depending on the service you're providing to me. Sufficient evidence may include my birth certificate or passport, an assortment of historical information, and other knowledge of the details that the government has stored about 'Jared'.
In other instances, social proof may be OK. The fact that others refer to me as 'Jared' may be sufficient evidence, and we can build up a body of evidence based on a history of social interactions.
Or perhaps you're a hitman paranoid about bumping off the wrong person, so you brute-force the solution. You stalk me for a month and rummage through my mailbox, finally whacking me on the head and rifling through the contents of my house and wallet.
Each one of these approaches may be required for sufficiency, and is appropriate for different contexts and purposes. And we might take similar approaches when we're testing software. We might simply talk to people who can provide evidence of a claim being met. We might perform a simple confirmatory test for a non-critical function. We might spend a longer time collecting a large body evidence to support a claim being met. It's important that we think about the importance of the claim and ensure that our approaches to gathering information can be defended under scrutiny.
The real-world analogy to the second point about claims testing - 'Expect the spec and the product to be brought into alignment' is perhaps more commonly found in legal circles, and occasionally in the forum posts or blog rants of less careful men who feel their masculinity has been threatened. It tends to involve situations like this. And as in a situation detailed here that more closely parallels software testing, remember that refuting a claim can sometimes have much bigger consequences.
I don't normally like to mix up entertainment with the software posts, but this one does have a vague relationship to the 'Puppy poo' story. It also invokes the PC Engine game of yore, 'Toilet kids'. And it's a nice interactive flash animation.
Sit back, relax, and enjoy Kkukku and Yaya in
'ppung ppung ppung eungga hagi'.
I found this while following a link to study resources on Korean Class 101. The free podcasts they have are excellent and the forums are helpful. The website family also has other languages in the same conversational style, in case you're looking for something other than Korean. You can find the podcasts via iTunes, or on their website. I've been enjoying their services for free, it's nice to give something back.
There have been a number of threads I have followed in a few different forums recently where people have discussed requirements, what it means for requirements to be 'good', and what it might mean for requirements to be unambiguous. What usually follows is a long-winded back and forth, with no resolution.
At the heart of this deja vu is the fact that one person's requirements are not necessarily another's. I resolve this by drawing a clear distinction between specifications and requirements. The distinction may be obvious to some, but in practice it seems to be something we struggle with.
In "Software for dependable systems: Sufficient evidence?", Daniel Jackson and his co-authors take care to highlight this issue:
They also take care to point out that the environment of the system includes the software product, plus the humans that use them, and other environmental factors external to the software.
The specifications are not about things that anybody *needs*. Specifications represent the end result of negotiations, conversations, politics and expediency that some group of people thinks represents an understanding that is good enough for now. The specification is at best, our best guess of what's going to make the world a better place for the numerous people who have a stake in the thing that we're building. Specifications are a waypoint on the path to something else.
Specifications are never equivalent to requirements in the case of things that will be used by humans.
Specifications apply to the pointy end of a screwdriver that needs to fit into the indented part of some screw.
As testers, testing to specifications is something we do because finding about the requirements is too hard. We do it because that's what the testers before us did. We do it because the process might be built around specifications documents, and that's what managers are tracking to. We might reasonably test to spec if our job is just to test a software component that's on its way to be integrated with something else. However, testing to spec can't tell us that the system is going to yield the desired benefits.
In the case of the screwdriver, requirements apply to the handle - how it fits your hand, and the hands of others. They apply to whether it gives you enough grip and whether it has a switch to make it only screw or unscrew. They apply to whether it fits enough hands in the world, and whether it can be built to a price that someone is willing to pay. Requirements apply to whether the pointy end of the screwdriver will make do for unscrewing (or screwing in) a screw that is not of the size covered by the specification for the pointy end of our screwdriver.
Requirements are about utility and real human problems, and are fuzzy, and messy, and never fully understood. When our stakeholders ask us questions about testing, though they often don't phrase it this way, they are usually interested in information with respect to requirements, both explicit and implicit.
Test to see how the product measures up to requirements, and to learn about what the requirements are. That's the value that you can bring to the project.
At the heart of this deja vu is the fact that one person's requirements are not necessarily another's. I resolve this by drawing a clear distinction between specifications and requirements. The distinction may be obvious to some, but in practice it seems to be something we struggle with.
In "Software for dependable systems: Sufficient evidence?", Daniel Jackson and his co-authors take care to highlight this issue:
"Software systems that are developed specially for a particular client are typically built to meet preagreed requirements, but these requirements are often a long and undifferentiated list of detailed functions."
"The requirements of a software system should describe the intended effect of the system on its environment and not, more narrowly, the behavior of the system at its interface with the environment, which is the subject of the specification."
They also take care to point out that the environment of the system includes the software product, plus the humans that use them, and other environmental factors external to the software.
The specifications are not about things that anybody *needs*. Specifications represent the end result of negotiations, conversations, politics and expediency that some group of people thinks represents an understanding that is good enough for now. The specification is at best, our best guess of what's going to make the world a better place for the numerous people who have a stake in the thing that we're building. Specifications are a waypoint on the path to something else.
Specifications are never equivalent to requirements in the case of things that will be used by humans.
Specifications apply to the pointy end of a screwdriver that needs to fit into the indented part of some screw.
As testers, testing to specifications is something we do because finding about the requirements is too hard. We do it because that's what the testers before us did. We do it because the process might be built around specifications documents, and that's what managers are tracking to. We might reasonably test to spec if our job is just to test a software component that's on its way to be integrated with something else. However, testing to spec can't tell us that the system is going to yield the desired benefits.
In the case of the screwdriver, requirements apply to the handle - how it fits your hand, and the hands of others. They apply to whether it gives you enough grip and whether it has a switch to make it only screw or unscrew. They apply to whether it fits enough hands in the world, and whether it can be built to a price that someone is willing to pay. Requirements apply to whether the pointy end of the screwdriver will make do for unscrewing (or screwing in) a screw that is not of the size covered by the specification for the pointy end of our screwdriver.
Requirements are about utility and real human problems, and are fuzzy, and messy, and never fully understood. When our stakeholders ask us questions about testing, though they often don't phrase it this way, they are usually interested in information with respect to requirements, both explicit and implicit.
Test to see how the product measures up to requirements, and to learn about what the requirements are. That's the value that you can bring to the project.
Otherwise, I might not be able to remember to buy it later.
19/08: Test for Project Risk
An excellent quote from our development lead, James Ladd:
How to test a project might be risky -
- It has people in it.
Thanks to James Bach for inspring this!
How to test a project might be risky -
- It has people in it.
Thanks to James Bach for inspring this!
"In the universe, nothing can be said to be automatic, as nothing can be said to be without design. An imperfect parallel may be found in human inventions; springs may move springs, and wheels, indexes; but the motion and the regulation must be derived from the artist;"
From Elements of Chemical Philosophy Part 1, Vol.1 By Humphry Davy
Of course, the second step will be taking my automation or testing tools course (plug, plug).
Paul Szymkowiak and I will be running a one-day workshop on tools for testers, as part of the STANZ conference. The current run on August 13th is full, but you can register your interest for future courses via SoftEd, or drop me a line.
Details of the course are at http://www.softed.com/stanz/speakersandsessions.htm#tst
I'll have the presentation slides here on the website shortly!
Details of the course are at http://www.softed.com/stanz/speakersandsessions.htm#tst
I'll have the presentation slides here on the website shortly!
16/07: Answering a question...
A few weeks ago, Designer commented on Software testing, art and productivity. The question got lost in amongst the comment spam, so I thought I'd give my answer a bit more prominently than usual. The question was:
If you're talking about smaller web-based projects, I think you're right. It can be really difficult to engage clients in the necessary up-front work to help reduce project uncertainty to a reasonable level. It's a problem given that those with less money to spend have much more business risk in any project they undertake.
I think the second aspect of this is that customers (both internal and external) often come to us with a solution, not a problem. As we build the solution they asked for, the problem becomes clearer and dissatisfaction starts to creep in.
The focus of my work is increasingly on trying to help people build the right thing in the first place. I'm lucky. In my current role, my employer has the conviction that it's important to make sure the project is heading in the right direction. This means making sure the project team has a shared understanding of the product vision, stakholders, those stakeholders' goals, priorities and (in the case of a consumer product) the market and opportunities. They also think that it's OK to bring development to a pause while we get our project bearings.
If you can't choose the projects you undertake, then I don't see any easy answers to these problems. And if you can't convince your customer to be involved appropriately, and to place *some* value on just talking about the problem, it's a hard road ahead for everyone. I guess the desire to do things 'right' is what drives many of us to start our own companies and projects. Without this option though, we can still focus on providing service - helping our customers better understand their problems and pointing out the benefits, costs and risks present in their solution(s). But choose your moments well, and don't stop dreaming that things can be better than they are.
...Many people who want to get a web-developed project don't even understand the details of work. They just want to have a result and not to make a lot of efforts. How do you think - is there any solution? I think it is wide-spread problem.
If you're talking about smaller web-based projects, I think you're right. It can be really difficult to engage clients in the necessary up-front work to help reduce project uncertainty to a reasonable level. It's a problem given that those with less money to spend have much more business risk in any project they undertake.
I think the second aspect of this is that customers (both internal and external) often come to us with a solution, not a problem. As we build the solution they asked for, the problem becomes clearer and dissatisfaction starts to creep in.
The focus of my work is increasingly on trying to help people build the right thing in the first place. I'm lucky. In my current role, my employer has the conviction that it's important to make sure the project is heading in the right direction. This means making sure the project team has a shared understanding of the product vision, stakholders, those stakeholders' goals, priorities and (in the case of a consumer product) the market and opportunities. They also think that it's OK to bring development to a pause while we get our project bearings.
If you can't choose the projects you undertake, then I don't see any easy answers to these problems. And if you can't convince your customer to be involved appropriately, and to place *some* value on just talking about the problem, it's a hard road ahead for everyone. I guess the desire to do things 'right' is what drives many of us to start our own companies and projects. Without this option though, we can still focus on providing service - helping our customers better understand their problems and pointing out the benefits, costs and risks present in their solution(s). But choose your moments well, and don't stop dreaming that things can be better than they are.
Mike Cohn's "User Stories Applied" discusses using the INVEST mnemonic as a guide to writing better user stories. I was recently asked to dig up a reference for it, and found this presentation here, with the section on the mnemonic on pages 47 and 48.
As I read it, I noticed that there's been a change to one of the letters. Whereas the book uses 'S' to denote 'Small', now it's become 'Sized appropriately'.
I think this is a change for the better, as I noticed that every time I talked someone through the acronym, I would have a long-winded conversation qualifying 'Small' as 'Just small enough but no smaller'. This would come about as I tried to explain the tradeoffs between a story being small enough to estimate with some reasonable certainty, small enough to fit within an iteration, and still ensuring that stories are 'V- Valuable to the customer', needing to ensure that user stories continue to express clearly a problem or need of a person.
Overly small stories push us further from the original context of the problem, and thus force us to compensate with an increasing heirarchy of 'super-stories' to help us focus on the bigger picture. These become more noticable when working in shorter iterations than might be common on a Scrum project, so three cheers for Mike in spending the effort to come up with 'Sized appropriately'.
As I read it, I noticed that there's been a change to one of the letters. Whereas the book uses 'S' to denote 'Small', now it's become 'Sized appropriately'.
I think this is a change for the better, as I noticed that every time I talked someone through the acronym, I would have a long-winded conversation qualifying 'Small' as 'Just small enough but no smaller'. This would come about as I tried to explain the tradeoffs between a story being small enough to estimate with some reasonable certainty, small enough to fit within an iteration, and still ensuring that stories are 'V- Valuable to the customer', needing to ensure that user stories continue to express clearly a problem or need of a person.
Overly small stories push us further from the original context of the problem, and thus force us to compensate with an increasing heirarchy of 'super-stories' to help us focus on the bigger picture. These become more noticable when working in shorter iterations than might be common on a Scrum project, so three cheers for Mike in spending the effort to come up with 'Sized appropriately'.
RSS