Anything your heart desires. And plenty that it won't. Ever.
It's a virtual smorgasbord of technologies, a couple of after-dinner stacks, and an utter testing nightmare. If it weren't so hellish, it'd be funny.
So here's an apology in advance for a thousand questions and no answers: I'm sorry.
How do you go about testing such a system?
First, a little background. It's not that
bad. We're not talking about the monolithic beast collapsing under its own weight. It is a monolithic beast, but a fairly stable one. Robustness could be improved.
Sharepoint makes up the bulk of the interface with users. It stores a lot of the data, some of which
changes often, some of which rarely does. It serves as
the shell for reports through Reporting Services, applications built in Ruby on Rails and ASP.NET, and third party plugins.
Because of the judicious use of these applications, most of the code is already well tested, and that code generates the rest. It's the seams that tend to break - rarely the units.
AD serves as the users database. It tracks a lot of information about users, and clearly, takes care of most authentication and access rights.
Most of the heavy lifting is taken care of in the database. That's because it's a very data-centric set of applications. It's much more involved than CRUD
Several stored procedures use multiple views, and are consumed by multiple parties. A change might work in one system, but it's impossible to know which others ones it affects.
The way all of these technologies integrate and serve the users is truly a thing of beauty. From the outside. From the inside, it often resembles a hideous monster, though not always.
Finally, although the team is forced to basically make all changes to a production system, they are lucky enough to have ways where it is not visible to the users until the changes are complete (or else, the changes are instantaneous and because of the technologies in use, well tested unit-wise, by default).
It seems clear to me that even if you could just catch (potential) bugs soon
after they are inserted, it would be a huge win. Is it even possible to catch them before hand?
So what would you do? Where would you even start?
I have some ideas. I'd like to hear yours as well though.
Log and Categorize Errors and Support Requests
First, instead of fixing errors as they are discovered and handling support requests as they come in, the team should log and categorize them. With that body of knowledge, you can apply measurement to determine where the worst offenders reside. Start there. Kill mercilessly. It gives an idea of where you can focus effort to get the highest return on your investment.
Trap and Report Errors as Much and as Soon as Possible
One easy thing that would take care of most of the .NET code would be to set up an error reporting script in the root Global.asax file. That will report crashes everywhere. I'm not convinced there are that many actual unhandled exceptions, but it's easy enough to insert in 5 minutes, and gets you sorely needed information immediately. Unfortunately, errors based on DB schema reported through Reporting Services are handled there - so I don't think they'll filter down to Global.asax.
It might be worth it to set up simple tests that hit the stored procedures in the DB often, just to see if they're still returning the expected columns and types that it sent back in the past. Alert the developers if not. If simple errors like that made it into production, they'll be found and corrected quickly. Simple errors like that are easy to make when multiple points hit the same spot - the one you were changing for use in A but that gets consumed by B, C, ... and Z.
A Complete Replica Test Bed
Ideally, eventually there would be a complete replica test environment set up. It has to be complete. The major point of such a system is it's integration capabilities, and that's where it seems (without having measured yet) the difficulties present themselves.
Disregarding the many
man-hours it would take to set this environment up, its existence is not enough because there currently exists no way to automatically push changes from a test environment to the production version.
Without that, this part of the plan is useless because the viscosity caused by having to make each change twice will inevitably lead to making it only once, which gets us back to where we started.
Scriptable In-Browser Tests
While I've been a fan of Selenium
in the past (and still am, to some extent), I also realize how fragile that type of testing is. In a case such as the one I've presented, I wonder if the upkeep of the tests will outweigh the benefits of having them. Right now I'm leaning towards useless. However, I can be convinced otherwise.
I'd need to dig deeper into it though. I don't know that I've used Selenium hands-free in the past.
The Automated / Human-Driven Testing Divide
As much of the testing as possible needs to be automated. The set of applications is far to big to be testing everything by hand every time something changes or gets added. That said, I understand and expect that some hand-testing will be necessary.
As yet unnamed tools which may or may not exist
A spider which runs under multiple accounts, asks once for classes of input and continuously runs through the app, but which can pass a value for "just kidding, don't really save that." What's out there?
Wherein I Ask Again, What Would You Do?
Here in the pragmatic real world, suggestions such as "throw
it away and write it from scratch as a single system" don't work. All other feedback is welcome, encouraged, and sought after.
What would you ask? Where would you look? What ideas, no matter how inane or simple they seem, might you try? Experts and newbies are all welcome. I don't know if anyone
has the answers, so all
feedback is encouraged. Think of it as a puzzle you're solving, not a "I'm not an expert so I better shut up" life or death situation.
Thanks in advance.
Hey! Why don't you make your life easier and subscribe to the full post
or short blurb RSS feed? I'm so confident you'll love my smelly pasta plate
wisdom that I'm offering a no-strings-attached, lifetime money back guarantee!
Leave a comment
To whom it may concern can i please use the picture under the quote "So what would you do? Where would you even start?" as a temp band cover image? Please get back to me via email (firstname.lastname@example.org)
Posted by Lauren Eady
on May 25, 2010 at 04:49 AM UTC - 5 hrs
Leave a comment
To all inquiring about the screaming picture above:
I wish I could say I knew who the original artist was. I've tried to get into the habit of linking to the source where I found the image, but in this case, I'm just not sure, so you'd need to find the artist and ask them.
I don't remember what I searched for when I found the image, but I did take the time to sign up for an account on tineye.com, which searches similar images from around the web, and have a link where you might start looking for the artist:http://www.tineye.com/search/71d1c56a143b244926bb5...
A lot of those links aren't to the actual page the artwork appeared on, but to the front page of the site instead.
Hopefully that will be enough to let you find them.
If you do find out who the artist is, and where I could link to their site, would you do me the favor of letting me know?
Posted by Sammy Larbi
on May 25, 2010 at 07:39 AM UTC - 5 hrs