Active Directory. Sharepoint. Reporting Services. External DBs. Nightly updates to untold numbers of internal DBs, spread among 4 DB servers. Ruby on Rails, ASP.NET, and JavaScript to customize specific Sharepoint pages. Apps in iframes and apps deployed as web parts. We've got stored procs and views upon Sharepoint's awesome Godtable, AllUserData. (Awesome - as in, "inspiring awe," not "totally rad dude!")

Anything your heart desires. And plenty that it won't. Ever.

It's a virtual smorgasbord of technologies, a couple of after-dinner stacks, and an utter testing nightmare. If it weren't so hellish, it'd be funny.



So here's an apology in advance for a thousand questions and no answers: I'm sorry.

How do you go about testing such a system?

First, a little background. It's not that bad. We're not talking about the monolithic beast collapsing under its own weight. It is a monolithic beast, but a fairly stable one. Robustness could be improved.

Sharepoint makes up the bulk of the interface with users. It stores a lot of the data, some of which changes often, some of which rarely does. It serves as the shell for reports through Reporting Services, applications built in Ruby on Rails and ASP.NET, and third party plugins. Because of the judicious use of these applications, most of the code is already well tested, and that code generates the rest. It's the seams that tend to break - rarely the units.

The JavaScript in individual Sharepoint pages is extremely fragile. Changes to the schema, data, or even field sizes can break it. Such is the nature of the customizations.

AD serves as the users database. It tracks a lot of information about users, and clearly, takes care of most authentication and access rights.

Most of the heavy lifting is taken care of in the database. That's because it's a very data-centric set of applications. It's much more involved than CRUD though.

Several stored procedures use multiple views, and are consumed by multiple parties. A change might work in one system, but it's impossible to know which others ones it affects.

The way all of these technologies integrate and serve the users is truly a thing of beauty. From the outside. From the inside, it often resembles a hideous monster, though not always.

Finally, although the team is forced to basically make all changes to a production system, they are lucky enough to have ways where it is not visible to the users until the changes are complete (or else, the changes are instantaneous and because of the technologies in use, well tested unit-wise, by default).

It seems clear to me that even if you could just catch (potential) bugs soon after they are inserted, it would be a huge win. Is it even possible to catch them before hand?

So what would you do? Where would you even start?



I have some ideas. I'd like to hear yours as well though.

Log and Categorize Errors and Support Requests
First, instead of fixing errors as they are discovered and handling support requests as they come in, the team should log and categorize them. With that body of knowledge, you can apply measurement to determine where the worst offenders reside. Start there. Kill mercilessly. It gives an idea of where you can focus effort to get the highest return on your investment.

Trap and Report Errors as Much and as Soon as Possible
One easy thing that would take care of most of the .NET code would be to set up an error reporting script in the root Global.asax file. That will report crashes everywhere. I'm not convinced there are that many actual unhandled exceptions, but it's easy enough to insert in 5 minutes, and gets you sorely needed information immediately. Unfortunately, errors based on DB schema reported through Reporting Services are handled there - so I don't think they'll filter down to Global.asax.

It might be worth it to set up simple tests that hit the stored procedures in the DB often, just to see if they're still returning the expected columns and types that it sent back in the past. Alert the developers if not. If simple errors like that made it into production, they'll be found and corrected quickly. Simple errors like that are easy to make when multiple points hit the same spot - the one you were changing for use in A but that gets consumed by B, C, ... and Z.

A Complete Replica Test Bed
Ideally, eventually there would be a complete replica test environment set up. It has to be complete. The major point of such a system is it's integration capabilities, and that's where it seems (without having measured yet) the difficulties present themselves.

Disregarding the many man-hours it would take to set this environment up, its existence is not enough because there currently exists no way to automatically push changes from a test environment to the production version. Without that, this part of the plan is useless because the viscosity caused by having to make each change twice will inevitably lead to making it only once, which gets us back to where we started.

Scriptable In-Browser Tests
While I've been a fan of Selenium in the past (and still am, to some extent), I also realize how fragile that type of testing is. In a case such as the one I've presented, I wonder if the upkeep of the tests will outweigh the benefits of having them. Right now I'm leaning towards useless. However, I can be convinced otherwise.

I'd need to dig deeper into it though. I don't know that I've used Selenium hands-free in the past.

The Automated / Human-Driven Testing Divide
As much of the testing as possible needs to be automated. The set of applications is far to big to be testing everything by hand every time something changes or gets added. That said, I understand and expect that some hand-testing will be necessary.

As yet unnamed tools which may or may not exist
A spider which runs under multiple accounts, asks once for classes of input and continuously runs through the app, but which can pass a value for "just kidding, don't really save that." What's out there?

Wherein I Ask Again, What Would You Do?
Here in the pragmatic real world, suggestions such as "throw it away and write it from scratch as a single system" don't work. All other feedback is welcome, encouraged, and sought after.

What would you ask? Where would you look? What ideas, no matter how inane or simple they seem, might you try? Experts and newbies are all welcome. I don't know if anyone has the answers, so all feedback is encouraged. Think of it as a puzzle you're solving, not a "I'm not an expert so I better shut up" life or death situation.

Thanks in advance.

Hey! Why don't you make your life easier and subscribe to the full post or short blurb RSS feed? I'm so confident you'll love my smelly pasta plate wisdom that I'm offering a no-strings-attached, lifetime money back guarantee!

Leave a comment



Leave a comment

Leave this field empty

	Your Name
	Email (not displayed, more info?) Privacy Information x Your email address is not displayed. It is used only to respond to you if needed, and send comments if you subscribe to this comment thread. It is stored in a cookie if you choose to "Remember my details".
	Website

Comment:

Subcribe to this comment thread
Remember my details