Testing MRMap - The Philosophy
Posted: Tue Jan 20, 2009 12:11 pm
In conversation with one of our program testers I described how I go about getting MRMap releases put through their paces before letting them loose on the rest of the MR world. In case anyone is interested, this is the philosophy behind it (Just to prove that there is one!)
Dave Binks is a professional software engineer and writes code to the level you would expect. However although he tests it before I get hold of it, I then handle some of the routine user interface and general 'funnies' testing. Even this takes some time so in order that the releases come out to yourselves as quickly as possible, I sub out the testing to a group of people who have been there pretty much from the outset and we all give the current release a good going over before you get it.
Initially we are looking for routine code bugs, things that crash the program possibly because of things that live on different computers that Dave has no way of predicting will have that effect. We're not a commecial enterprise in any way so we have to do all of this ourselves and we don't have access to the number of computers running all kinds of software that maybe the industry has. Because of this I have a group of 'volunteers' all over England & Wales, Northern Ireland and the Republic of Ireland who have at some time in their lives written computer code at one level or another. They aren't required to write any code for us (although some do) but what they do know about is how programs are broken by users who just don't do what the developer wants or expects them to do. One of the new functions coming out in the next release is the ability to individually colour snail trails, it requires an RGB colour value to be entered where three numbers none of which can be smaller than 0 or bigger than 255 are used. Of course, the testers enter -255 or 999 just to try and break the code just as I do. We all do absolutely the best we can to wreck Daves efforts and crash the program. In fact he saw this one coming and we couldn't break it.
This is testing at a purely functional 'Will it crash?' level. Depending on how many new functions have been added to a new release, and there are at least three in the next one, I then pass the new version on to a second set of testers. These are testing the operational effects and efficiency of any added functions. In some cases they are the same people who can test for pure programming errors but not always. These ladies and gentlemen are time-served mountain rescue. Usually team leaders or deputies or at least they have been at some stage in their lives. They are the ones who guide MRMap along its path to operational efficiency and usefulness. If they don't like the way something has been implemented, they will say so and we either have to argue the case for how it's been done, or Dave re-writes the code to satisfy the tester. In one such case, myself and another tester disagreed as to whether the marker that you can place on the map, should be a circle or a flag. We couldn't resolve that one so Dave added an ini file value that will allow you, the user, to make your own choice. Virtually everything you see in MRMap today has gone through that process.
Not all the available testers get the next release each time as there are hopefully enough changes every couple of releases to avoid the danger of them thinking, 'Not much has changed and the last release worked, so this one is probably OK'. We never get blase about the testing and hopefully each release Dave puts out has been sujected to some very rigorous processes before you get hold of it.
Our aim is that there should be no testing phase required by yourselves. We do our best to ensure that each release is actually useable straight out of the box. We're not always right but we've been right more times than we're wrong! By and large you can take a new release of MRMap and plug it straight in. This whole exercise hinges on feedback and for that I have a third wave of testers, yourselves. We do need your feedback however and I would ask you all to not sit on or suffer anything you don't like or think doesn't work. Just contact me with as much detail as you can and we will look into it.
Suggestions for improvement are never 'stupid' if you see them as being of use or advantage to your team. What might get them (politely) rejected is that they clash with another function already in the program or that they increase the complexity of the program beyond that which permits the current level and methods of development to continue. Our policy is to keep the program simple to use and some requests would require a significant level of user interaction that goes against this thinking. This is the reason why the ini file is what it is. Mostly you don't have to touch it and the program will be happy to operate at it's basic level without needing anything from you. However there are some things you can modify which might make life easier for your team and the user of MRMap. This modification process, via the ini file, is kept hidden from normal everyday use but is there if you feel you can cope with it.
This is the reason why there's an MRMap User Manual that to just get the program running, you don't even need to open. The ladies amongst our ranks will immediately recognise this as a 'Man thing' I'm sure. Having been guilty on numerous occasions of doing things without first reading the manual, I decided to take this a stage further and following Dave's philosophy of keeping MRMap simple, I wrote a user manual you don't need to read until there's something a level or two up from 'out-of-the-box' usage that you want to try.
So that's pretty much how it's done. I continue to owe a debt of thanks to all the people who give up yet more of their time to help us in this way. Most of you know who you are, those who suddenly get a copy of the next version of MRMap out of the blue, now know why! Either way, very many thanks to you all, we couldn't do it without you.
The MRMap Development Team
Dave Binks is a professional software engineer and writes code to the level you would expect. However although he tests it before I get hold of it, I then handle some of the routine user interface and general 'funnies' testing. Even this takes some time so in order that the releases come out to yourselves as quickly as possible, I sub out the testing to a group of people who have been there pretty much from the outset and we all give the current release a good going over before you get it.
Initially we are looking for routine code bugs, things that crash the program possibly because of things that live on different computers that Dave has no way of predicting will have that effect. We're not a commecial enterprise in any way so we have to do all of this ourselves and we don't have access to the number of computers running all kinds of software that maybe the industry has. Because of this I have a group of 'volunteers' all over England & Wales, Northern Ireland and the Republic of Ireland who have at some time in their lives written computer code at one level or another. They aren't required to write any code for us (although some do) but what they do know about is how programs are broken by users who just don't do what the developer wants or expects them to do. One of the new functions coming out in the next release is the ability to individually colour snail trails, it requires an RGB colour value to be entered where three numbers none of which can be smaller than 0 or bigger than 255 are used. Of course, the testers enter -255 or 999 just to try and break the code just as I do. We all do absolutely the best we can to wreck Daves efforts and crash the program. In fact he saw this one coming and we couldn't break it.
This is testing at a purely functional 'Will it crash?' level. Depending on how many new functions have been added to a new release, and there are at least three in the next one, I then pass the new version on to a second set of testers. These are testing the operational effects and efficiency of any added functions. In some cases they are the same people who can test for pure programming errors but not always. These ladies and gentlemen are time-served mountain rescue. Usually team leaders or deputies or at least they have been at some stage in their lives. They are the ones who guide MRMap along its path to operational efficiency and usefulness. If they don't like the way something has been implemented, they will say so and we either have to argue the case for how it's been done, or Dave re-writes the code to satisfy the tester. In one such case, myself and another tester disagreed as to whether the marker that you can place on the map, should be a circle or a flag. We couldn't resolve that one so Dave added an ini file value that will allow you, the user, to make your own choice. Virtually everything you see in MRMap today has gone through that process.
Not all the available testers get the next release each time as there are hopefully enough changes every couple of releases to avoid the danger of them thinking, 'Not much has changed and the last release worked, so this one is probably OK'. We never get blase about the testing and hopefully each release Dave puts out has been sujected to some very rigorous processes before you get hold of it.
Our aim is that there should be no testing phase required by yourselves. We do our best to ensure that each release is actually useable straight out of the box. We're not always right but we've been right more times than we're wrong! By and large you can take a new release of MRMap and plug it straight in. This whole exercise hinges on feedback and for that I have a third wave of testers, yourselves. We do need your feedback however and I would ask you all to not sit on or suffer anything you don't like or think doesn't work. Just contact me with as much detail as you can and we will look into it.
Suggestions for improvement are never 'stupid' if you see them as being of use or advantage to your team. What might get them (politely) rejected is that they clash with another function already in the program or that they increase the complexity of the program beyond that which permits the current level and methods of development to continue. Our policy is to keep the program simple to use and some requests would require a significant level of user interaction that goes against this thinking. This is the reason why the ini file is what it is. Mostly you don't have to touch it and the program will be happy to operate at it's basic level without needing anything from you. However there are some things you can modify which might make life easier for your team and the user of MRMap. This modification process, via the ini file, is kept hidden from normal everyday use but is there if you feel you can cope with it.
This is the reason why there's an MRMap User Manual that to just get the program running, you don't even need to open. The ladies amongst our ranks will immediately recognise this as a 'Man thing' I'm sure. Having been guilty on numerous occasions of doing things without first reading the manual, I decided to take this a stage further and following Dave's philosophy of keeping MRMap simple, I wrote a user manual you don't need to read until there's something a level or two up from 'out-of-the-box' usage that you want to try.
So that's pretty much how it's done. I continue to owe a debt of thanks to all the people who give up yet more of their time to help us in this way. Most of you know who you are, those who suddenly get a copy of the next version of MRMap out of the blue, now know why! Either way, very many thanks to you all, we couldn't do it without you.
The MRMap Development Team