• What kind of tools are important to you for monitoring a product during maintenance?
  I rely on logs and profiling tools on occasion. I'm really interested to hear from the rest of you about this though.
  
  
• What is important when updating a product that is in production and is being used?
  I'd say it's important not to interrupt service. Surely there must be something else you're getting at?
  
  
• How do you find an error in a large file with code that you cannot step through?
  cout, puts, printf, System.Out.print, Console.Out.WriteLine, and ## have all been useful for me at one time or another.
  
  A good strategy here is to isolate the code that's causing the error by removing code and faking results. By doing that, you can slowly add code back in until the error reappears. Rewrite that part.
  
  
• How can you make sure that changes in code will not affect any other parts of the product?
  Regression tests!
  
  
• How do you create technical documentation for your products?
  Ideally I'd have comments that can be harvested by tools like RDoc or JavaDoc, but times are often less than ideal.
  
  
  
  
• What measures have you taken to make your software products more easily maintainable?
  See my answers to the questions about technical design. Also, having unit tests helps handily.
  
  
• How can you debug a system in a production environment, while it is being used?
  You can read logs if important events are being logged. Profiling tools exist for this purpose, but I don't have experience with any outside of those for use with databases.
  
  
• Do you know what load balancing is? Can you name different types of load balancing?
  One computer acts as the gatekeeper for an array of computers and directs requests to the others to "balance the load" of the entire system.
  
  I'm not familiar with different types, but just guessing I'd assume they have round-robin and need-based load balancing. I'd also presume any other scheduling algorithmic scheme could be applied in load balancing.
  
  I'm more interested to know why this is on the maintenance list of questions. Is it because you've deployed your application and now you need to scale it with hardware?
  
  
• Can you name reasons why maintenance of software is the biggest/most expensive part of an application's life cycle?
  One view is that after you write the first line of code, you begin maintenance. But more in-line with the popular view: it lasts the longest. You may take a month to build a system that will be in production over several years. During that time, defects are found that need to be fixed, business rules may change, or new features may be added.
  
  Also, we suck at writing software.
  
  
  
  
• What is the difference between re-engineering and reverse engineering?
  I didn't know this one. I thought and would have responded that re-engineering would be rebuilding an application with a white box, while reverse engineering would be done through a black box.
  
  According to Wikipedia, who is never wrong,

  The reengineering of software was described by Chikofsky and Cross in their 1990 paper, as "The examination and alteration of a system to reconstitute it in a new form". Less formally, reengineering is the modification of a software system that takes place after it has been reverse engineered, generally to add new functionality, or to correct errors.
  
  This entire process is often erroneously referred to as reverse engineering; however, it is more accurate to say that reverse engineering is the initial examination of the system, and reengineering is the subsequent modification.

How would you answer these questions about software maintenance?

I think the solution is to start under-promising and over-delivering, as opposed to how most of us do it now: giving lowball estimates because we think that's what they want to hear. But why lie to them?

If you're using iterative and incremental development, then if you've over-promised one iteration, you are supposed to dial down your estimates for what you can accomplish in subsequent iterations, until you finally get good at estimating. And estimates should include what it takes to do it right.

That's the party-line answer to the question. In short: it's never OK to write sloppy code, and you should take precautions against ever putting yourself in a situation where those viscous forces pull you in that direction.

The party-line answer is the best answer, but it doesn't fully address the question, and I'm not always interested in party-line answers anyway. The viscosity (when it's easier to do the wrong thing that the right thing) is the force behind the bodge. I don't like it, but I recognize that there are going to be times you run into it and can't resist.

In those cases where you've already painted yourself into a corner, what then? That's the interesting question here. How do you know the best places to hack crapcode together and ignore those things that may take a little longer in the short run, but whose value shows up in the long run?

The easy answer is the obvious one: cut corners in the code that is least likely to need to change or be touched again. That's because (assuming your hack works) if we don't have to look at the code again, who really cares that it was a nasty hack? The question whose answer is not so easy or obvious is "what does such a place in the code look like?"

By the definition above, it would be the lower levels of your code. But if you do that, and inject a bug, then many other parts of your application would be affected. So maybe that's not the right place to do it. Instead, it would be better to do it in the higher levels, on which very little (if any) other code depends. That way, you limit the effects of it. More importantly, if there are no outgoing dependencies on it, it is easier to change than if other code were highly dependent on it. [1]

Maybe the crapcode can be isolated: if a class is already aweful, can you derive a new class from it and make any new additions with higher quality? If a class is of high quality and you need to hack something together, can you make a child class and put the hack there? [2]

Uncle Bob recently discussed when unit and acceptance testing may safely be done away with. He put those numbers around 10 and a few thousand lines of code, respectively.

In the end, there is no easy answer that I can find where I would definitively say, "that's the place for a bodging." But I suspect there are some patterns we can look for, and I tried to identify a couple of those above.

Do you have any candidates you'd like to share?

Notes:
[1] A passing thought for which I have no answers: The problem with even identifying those places is that by hacking together solutions, you are more likely to inject defects into the code, which makes it more likely you'll need to touch it again.

[2] I use inheritance here because the new classes should be able to be used without violating LSP. However, you may very well be able to make those changes by favoring composition. If you can, I'd advocate doing so.

The experiences discussed herein were valuable in their own right, but the challenge itself is rewarding as well. How often do we pause to reflect on what we've learned, and more importantly, how it has changed us? Because of that, I recommend you perform the exercise as well.

I freely admit that some of this isn't necessarily caused by my experiences with the language alone - but instead shaped by the languages and my experiences surrounding the times.

One last bit of administrata: Some of these memories are over a decade old, and therefore may bleed together and/or be unfactual. Please forgive the minor errors due to memory loss.


How QBASIC Made Me A Programmer

As I've said before, from the time I was very young, I had an interest in making games. I was enamored with my Atari 2600, and then later the NES. I also enjoyed a playground game with Donald Duck and Spelunker.

Before I was 10, I had a notepad with designs for my as-yet-unreleased blockbuster of a side-scrolling game that would run on my very own Super Sanola game console (I had the shell designed, not the electronics).

It was that intense interest in how to make a game that led me to inspect some of the source code Microsoft provided with QBASIC. After learning PRINT, INPUT, IF..THEN, and GOTO (and of course SomeLabel: to go to) I was ready to take a shot at my first text-based adventure game.

The game wasn't all that big - consisting of a few rooms, the NEWS directions, swinging of a sword against a few monsters, and keeping track of treasure and stats for everything - but it was a complete mess.

The experience with QBASIC taught me that, for any given program of sufficient complexity, you really only need three to four language constructs:

1. Input
2. Output
3. Conditional statements
4. Control structures

Even the control structures may not be necessary there. Why? Suppose you know a set of operations will be performed an unknown but arbitrary amount of times. Suppose also that it will be performed less than X number of times, where X is a known quantity smaller than infinity. Then you can simply write out X number of conditionals to cover all the cases. Not efficient, but not a requirement either.

Unfortunately, that experience and its lesson stuck with me for a while. (Hence, the title of this weblog.)

Side Note: The number of language constructs I mentioned that are necessary is not from a scientific source - just from the top of my head at the time I wrote it. If I'm wrong on the amount (be it too high or too low), I always appreciate corrections in the comments.


What ANSI Art taught me about programming

When I started making ANSI art, I was unaware of TheDraw. Instead, I opened up those .ans files I enjoyed looking at so much in MS-DOS Editor to see how it was done. A bunch of escape codes and blocks came together to produce a thing of visual beauty.



Since all I knew about were the escape codes and the blocks (alt-177, 178, 219-223 mostly), naturally I used the MS-DOS Editor to create my own art. The limitations of the medium were strangling, but that was what made it fun.

And I'm sure you can imagine the pain - worse than programming in an assembly language (at least for relatively small programs). Nevertheless, the experience taught me some valuable lessons:

• Even though we value people over tools, don't underestimate the value of a good tool. In fact, when attempting anything new to you, see if there's a tool that can help you. Back then, I was on local BBSs, and not the 1337 ones when I first started out. Now, the Internet is ubiquitous. We don't have an excuse anymore.
  
  
• I can now navigate through really bad code (and code that is limited by the language) a bit easier than I might otherwise have been able to do. I might have to do some experimenting to see what the symbols mean, but I imagine everyone would. And to be fair, I'm sure years of personally producing such crapcode also has something to do with my navigation abilities.
  
  
• Perhaps most importantly, it taught me the value of working in small chunks and taking baby steps. When you can't see the result of what you're doing, you've got to constantly check the results of the latest change, and most software systems are like that. Moreover, when you encounter something unexpected, an effective approach is to isolate the problem by isolating the code. In doing so, you can reproduce the problem and problem area, making the fix much easier.

The Middle Years (included for completeness' sake)

The middle years included exposure to Turbo Pascal, MASM, C, and C++, and some small experiences in other places as well. Although I learned many lessons, there are far too many to list here, and most are so small as to not be significant on their own. Therefore, they are uninteresting for the purposes of this post.

However, there were two lessons I learned from this time (but not during) that are significant:

1. Learn to compile your own $&*@%# programs (or, learn to fish instead of asking for them).
2. Stop being an arrogant know-it-all prick and admit you know nothing.

As you can tell, I was quite the cowboy coding young buck. I've tried to change that in recent years.


How ColdFusion made me a better programmer when I use Java

Although I've written a ton of bad code in ColdFusion, I've also written a couple of good lines here and there. I came into ColdFusion with the experiences I've related above this, and my early times with it definitely illustrate that fact. I cared nothing for small files, knew nothing of abstraction, and horrendous god-files were created as a result.

If you're a fan of Italian food, looking through my code would make your mouth water.

DRY principle? Forget about it. I still thought code reuse meant copy and paste.

Still, ColdFusion taught me one important aspect that got me started on the path to Object Oriented Enlightenment: Database access shouldn't require several lines of boilerplate code to execute one line of SQL.

Because of my experience with ColdFusion, I wrote my first reusable class in Java that took the boilerplating away, let me instantiate a single object, and use it for queries.


How Java taught me to write better programs in Ruby, C#, CF and others

It was around the time I started using Java quite a bit that I discovered Uncle Bob's Principles of OOD, so much of the improvement here is only indirectly related to Java.

Sure, I had heard about object oriented programming, but either I shrugged it off ("who needs that?") or didn't "get" it (or more likely, a combination of both).

Whatever it was, it took a couple of years of revisiting my own crapcode in ColdFusion and Java as a "professional" to tip me over the edge. I had to find a better way: Grad school here I come!

The better way was to find a new career. I was going to enter as a Political Scientist and drop programming altogether. I had seemingly lost all passion for the subject.

Fortunately for me now, the political science department wasn't accepting Spring entrance, so I decide to at least get started in computer science. Even more luckily, that first semester Venkat introduced me to the solution to many my problems, and got me excited about programming again.

I was using Java fairly heavily during all this time, so learning the principles behind OO in depth and in Java allowed me to extrapolate that for use in other languages. I focused on principles, not recipes.

On top of it all, Java taught me about unit testing with JUnit. Now, the first thing I look for when evaluating a language is a unit testing framework.


What Ruby taught me that the others didn't

My experience with Ruby over the last year or so has been invaluable. In particular, there are four lessons I've taken (or am in the process of taking):

• The importance of code as data, or higher-order functions, or first-order functions, or blocks or closures: After learning how to appropriately use yield, I really miss it when I'm using a language where it's lacking.
  
  
• There is value in viewing programming as the construction of lanugages, and DSLs are useful tools to have in your toolbox.
  
  
• Metaprogramming is OK. Before Ruby, I used metaprogramming very sparingly. Part of that is because I didn't understand it, and the other part is I didn't take the time to understand it because I had heard how slow it can make your programs.
  
  Needless to say, after seeing it in action in Ruby, I started using those features more extensively everywhere else. After seeing Rails, I very rarely write queries in ColdFusion - instead, I've got a component that takes care of it for me.
  
  
• Because of my interests in Java and Ruby, I've recently started browsing JRuby's source code and issue tracker. I'm not yet able to put into words what I'm learning, but that time will come with some more experience. In any case, I can't imagine that I'll learn nothing from the likes of Charlie Nutter, Ola Bini, Thomas Enebo, and others. Can you?

What's next?

Missing from my experience has been a functional language. Sure, I had a tiny bit of Lisp in college, but not enough to say I got anything out of it. So this year, I'm going to do something useful and not useful in Erlang. Perhaps next I'll go for Lisp. We'll see where time takes me after that.

That's been my journey. What's yours been like?

Now that I've written that post, I have a request for a post I'd like to see:

What have you learned from a non-programming-related discipline that's made you a better programmer?

1. JetBrains' IntelliJ IDEA: An awesome IDE for Java. So great, I don't mind spending the $249 (US) and using it over the free Eclipse. The Ruby plugin is not too shabby either, the license for your copy is good for your OSX and Windows installations, and you can try it free for 30 days. Martin Fowler thinks IntelliJ changed the IDE landscape. If you work in .NET, they also have ReSharper, which I plan to purchase very soon. Now if only we could get a ColdFusion plugin for IntelliJ, I'd love it even more.
   
   
2. Programming Ruby, Second Edition: What many in the Ruby community consider to be Ruby's Bible. You can lower the barrier of entry for your favorite programmer to using Ruby, certainly one of the funner languages a lot of people are loving to program in lately. Sometimes, I sit and think about things to program just so I can do it in Ruby.
   
   If they've already got that, I always like books as gifts. Some of my favorites from this year have been: Code Complete 2, Agile Software Development: Principles, Patterns, and Practices which has a great section on object oriented design principles, and of course, My Job Went to India. I have a slew of books I've yet to read this year that I got from last Christmas (and birthday), so I'll have to list those next year.
   
   
3. Xbox 360 and a subscription to XNA Creator's Club (through Xbox Live Marketplace - $99 anually) so they can deploy their games to their new Xbox. This is without a doubt the thing I'd want most, since I got into this whole programming thing because I was interested in making games. You can point them to the getting started page, and they could make games for the PC for free, using XNA (they'll need that page to get started anyway, even if you get them the 360 and Creator's Club membership to deploy to the Xbox).
   
   
4. MacBook Pro and pay for the extra pixels. I love mine - so much so, that I intend to marry it. (Ok, not that much, but I have been enjoying it.) The extra pixels make the screen almost as wide as two, and if you can get them an extra monitor I'd do that too. I've moved over to using this as my sole computer for development, and don't bother with the desktops at work or home anymore, except on rare occasions. You can run Windows on it, and the virtual machines are getting really good so that you ought not have to even reboot to use either operating system.
   
   Even if you don't want to get them the MacBook, a second or third monitor should be met with enthusiasm.
   
   
5. A Vacation: Programmers are notoriously working long hours and suffering burnout, so we often need to take a little break from the computer screen. I like SkyAuction because all the vacations are package deals, there's often a good variety to choose from (many different countries), most of the time you can find a very good price, and usually the dates are flexible within a certain time frame, so you don't have to commit right away to a certain date.

Happy Thanksgiving to those celebrating it, and thanks to all you who read and comment and set me straight when I'm wrong - not just here but in the community at large. I do appreciate it.

Do you have any ideas you'd like to share (or ones you'd like to strike from this list)?

If the parentheses are there to clarify a complex expression, that is one thing (but there are better ways to do it, such as using descriptive variable names). The parentheses add clutter, making the code ever-so-slightly harder to read.

Certainly no one would implement a scanner for string searching when a simple regular expression would do (I hope, anyway). It would be even more foolish if you did it just because the next coder to look at the code might find it easier to understand.

When I first read chromatic's funny phrase, I thought we should program as if those who follow us have unintelligible thoughts in their heads. (I still do.) I was looking at it like this:

If I write programs like a monkey would write programs (and I've been known to do such things), the person following me would need a highly developed brain to fit enough of it in his head to be of any use to the code he inherited. I don't even like to work on some of that old stuff, and I certainly don't understand half of it. Therefore, it would be better to write the code so that even a monkey can follow it. I'm not talking about concepts like Mark was talking about. Instead, I'm talking about things like five-line methods, cohesion, modularity, understandable code, and the sort.

I think both views work - they are just referring to different things.

Maintenance programmers get dumped on all the time. The least we could do is try to make their jobs easier by not feeding them crapcode. Maybe a Maintenance Programmer Appreciation Day is in order?

Update: Based on the discussion below, I made a couple of changes to avoid confusion above. Read the comments for more info on that discussion.

1. Know the fundamentals
   It was no surprise that Neal led off with the DRY principle. Being one from the school of thought that "code reuse" meant "copy-and-pastable with few changes," I feel this is the most important principle to follow when coding.
   
   Neal uses it in the original sense: Refuse to repeat yourself not just in your code, but all throughout the system. To quote (If I remember correctly, The Pragmatic Programmer): "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system." I can't think of anything that wouldn't be improved by following that advice.
   
   Continuous integration and version control are important fundamentals as well. Continuous integration means you know the code base at least compiles, and you can have it run your unit tests as well. The codebase is verified as always working (at least to the specs as you wrote them - you might have gotten them wrong, of course).
   
   Using version control means you don't have to be afraid of deleting that commented out code that you'll probably never use again but you keep around anyway "just in case." That keeps your code clean.
   
   Finally, Neal suggests we should use static analysis tools such as FindBugs and PMD that inspect our code, look for bad patterns, and suggest fixes and places to trim. Those two are for Java. Do you know of similar applications for other platforms? (Leave a comment please!)
   
   
2. Analyze code odors and remove them if they stink
   Inheritance is a smell. It doesn't always have to smell bad, but at least you should take the time to determine if what you really need is composition.
   
   Likewise, the existence of helper or utility classes often indicates a "poor design." As Neal notes, "if you've gotten the abstraction correct, why do you need 'helpers'?" Try to put them in "intelligent domain object[s]" instead.
   
   Static methods are an indication you are thinking procedurally, and they are "virtually impossible to test." Seeing the procedural mindset is easy, I think, but why are they hard to test?
   
   Finally, avoid singletons. I know Singleton is everyone's favorite design pattern because it's so easy to conceptualize, but as Neal mentions:
   • They're really just a poor excuse for global variables.
   • They violate SRP (link is to a PDF file) by mixing business logic with policing you from using too many
   • They make themselves hard to test
   • It's very hard to build a real singleton, as you have to ensure you are using a unified class loader.
   • More that I won't get into here...
   
   
3. Kill Sacred Cows
   Neal started off saying that "sacred cows make the tastiest hamburger" and told the story of the angry monkeys:
   
   A few monkeys were put in a room with a stepladder and a banana that could only be reached if the monkeys used the stepladder. However, when a monkey would climb upon the stepladder, all the monkeys would be doused with water. Then new monkeys were introduced and when they tried to use the stepladder, the other monkeys would get angry and beat him up. Eventually, the room contained none who had been doused with water, but the monkeys who had been beat up for using the stepladder were still refusing to allow new monkeys to get the banana.
   
   That is the light in which you may often view sacred cows in software development. In particular, Neal points out that
   • StickingToCamelCaseForSentenceLongTestNamesIsRidiculous so_it_is_ok_to_use_underscores in languages_that_standardized_on_camel_case for your test names.
   • Using getters and setters does not equate to encapsulation. One of my favorite sayings, and a reference to why they are evil is appropriate.
   • Avoidance of multiple returns in a method is antiquated because it comes from a time when we weren't using tiny, cohesive methods. I still like to avoid them, but will use them when it makes the code more readable.
   • Polluting interface names with "I", as in ISomeInterface should be avoided. It is contrary to what interfaces are about. Instead, you should decorate the concrete class name.
   
   
4. Your objects should be Good Citizens. Never let them exist in an invalid state.
   
   
5. Use Test Driven Development
   TDD provides "explicit dependency management" because you must think about dependencies as you code. It provides instant feedback and encourages you to implement the simplest thing that works.
   
   You can take it further by analyzing your dependencies with JDepend.
   
   And don't forget about YAGNI. Quoting Neal,

   Speculative development saves time if
   • You have nothing better to work on right now
   • You guarantee that it won't take longer to fix later

   Is that ever going to be true?
   
   
6. Use reflection when you can (and when it makes sense)
   It's not slow, and it can be extremely useful!
   
   
7. Colour Your World
   Nick Drew, a ThoughtWorker in the UK came up with a system of coloring the code you write by classifying it as to who will use the feature: only a specific business customer, only a particular market, many markets, or unchangeable infrastructure. Based on the color and amount of it, you can decide what value or cost your code has.
   
   It's a very interesting system, so I recommend seeing a more detailed overview in Neal's handout (PDF). You can find it on pages 21-22.
   
   
8. Use a DSL style of coding
   It lets you "[Build] better abstraction layers, using language instead of trees" while "utilizing" current building blocks. As he pointed out, "every complicated human endeavor has its own DSL. [If you can] make the code closer to the DSL of the business [then] it is better abstracted."
   
   I'll add that it's easier to spot logical errors as well.
   
   A funny quote from Neal: "It's almost as if in Java we're talking to a severely retarded person." Regarding that, he recommended looking at EasyMock as a good example of fluent interfaces, and that having setters return void in a waste. Instead, we should return this so that calls can be chained together (and if you are using fluent-interface type names, you could construct sentences in your DSL that way).
   
   Neal also noted a distinction between an API and DSL: API has an explicit context that must be repeated with each call. However, a DSL uses an implicit context.
   
   
9. SLAP tells us to keep all lines of code in a method at the same level of abstraction. Steve McConnell's Code Complete 2 tells us about this as well, but I don't recall if it had the clever acronym.
   
   
10. And finally, Think about Antiobjects.
    Quoting Neal, "Antiobjects are the inverse of what we perceive to be the computational objects." So instead of solving the really hard "foreground" problem, have a look at the "background" problem (the inverse) to see if it is easier to solve.
    
    As an example, he used PacMan: Rather than constructing a solution for the "shortest route around the maze," the game has a "notion of a PacMan 'scent'" for each tile." That way, the ghosts follow the strongest scent.

As with my write-ups on Scott Davis's Groovy: Greasing the Wheels of Java, and Stuart Halloway's JavaScript for Ajax Programmers, Neal gets all the credit for the good stuff here. I'm just reporting from my notes, his mindmap, and my memory, so any mistakes you find are probably mine. If they aren't, they probably should have been.

So which strategies have you used? Will you start using any of the above?

The first item where I see a distinction that needs to be made is on, "what do we mean when we are talking about XML?" I see two types - XML the paradigm where you use tags to describe data, and the XML you write - as in, the concrete tags you put into a file (like, "see that XML?"). We're talking about XML you've written, not the abstract notion of XML.

The second idea: what is code? What is data? Sean Corfield offers what I would consider to be a concice, and mostly correct definition: "Code executes, non-code (data) does not execute." To make it correct (rather than partially so), he adds that (especially in Lisp) code can be data, but data is not code. You see this code-as-data any time you are using closures or passing code around as data. But taking it a bit further - your source code is always just data to be passed to a compiler or interpreter, which figures out what the code means, and does what it has been told to do.

So is XML code? Certainly we see it can be: ColdFusion, MXML, and others are languages where your source code is written (largely) in XML. But what about the broader issue of having a programmatic configuration file versus a "data-only" config file?

Is the data in the config file executable? It depends on the purpose behind the data. In the case of data like


I think (assuming there is nothing strange going on) that it is clearly data. Without knowing anything about the back end, it seems like we're just creating a data structure. But In the case of DI (and many others uses for config files), I see it as giving a command to the DI framework to configure a new bean. In essence, as Peter notes, we've just substituted one concrete syntax for another.

In the case of XML, we're writing (or using) a parser to send data to an intepreter we've written that figures out what "real" commands to run based on what the programmer wrote in the configuration file. We've just created a higher level language than we had before - it is doing the same thing any other non-machine code language does (and you might even argue about the machine code comment). In the configuration case, often it is a DSL (in the DI case specifically, used to describe which objects depend on which other objects and load them for us).

While we don't often have control structures, there is nothing stopping us from implementing them, and as Peter also notes, just because a language is not Turing complete), doesn't mean it is not a programming language. In the end, I see it as code.

Both approaches are known to have their benefits and drawbacks, and choosing one over the other is largely a matter of personal taste, size and scope of problem, and problem/solution domain. For me, in the worlds of JIT compiling and interpreted langages, the programmatic way of doing things tends to win out - especially with large configurations because I prefer to have the power of control structures to help me do my job (without having to implement them myself). On the other hand, going the hard-coded XML route is especially popular in the compiled world, if not for any other reason than you can change configurations without recompiling your application.

I don't make a distinction between the two on terms of XML is data, while programming (or using an in-language DSL) in your general-purpose language is code. To me, they are both code, and if either is done incorrectly it will blow-up your program.

Finally, I'm not sure what value we gain from seeing that code is data (and in many cases config data is code), other than perhaps a new way of looking at problems which might lead us to find better solutions. But that isn't provided by the distinction itself, just the fact that we saw it.

Comments, thoughts, questions, and requests for clarifications are welcome and appreciated.

Initially, that struck me as an idea worth exploring, since those are two of my most-used, favorite, and life-saving principles (I mean, try violating them over and over for years and see if you don't feel like dying!). In fact, I use them a lot in determining when to create a new method (I use other heuristics as well, such as readability, which is often the most important reason I create a function ... more here about some reasons for creating methods). But, I thought about it some and came to this conclusion I posted to the pragprog group (which I changed a bit for here to make more sense):

In essence, I'm not comfortable with either of those definitions (DRY or YAGNI) as lines for determining when you are soft or hard coding. Suppose the following (generalizing to your pleasure -- since I'm having trouble writing it, I'm sure others will have trouble understanding it, as no one would ever (I think) write an application this way)

1. it always costs the same to ship each item: 5.00 (units of currency)
2. the company only needs to worry about one tax rate: 5%
3. the product costs 100 dollars each, and there is only one product.
4. customers can buy a variable quantity of this item (called "quantity" below)
5. there is a handling charge of 3.14 (units of currency) for each item.

Then your total_cost = (100+5+3.14)*(1.0+5/100)*quantity.

We see 100 twice here. Do we pull it out? What do we name it? oneHundred? That is the only one that makes sense for both cases. Or we could change it to to get rid of calculating the percentage: total_cost = (100+5+3.14)*(1.05)*quantity.

I'm not even confident I've got the right formula - and I'm the one making this up. What happens down the line if the state changes the sales tax? Am I sure I know which number to manipulate? Here, it may look clear, but it may not be the case in other situations. And then when I need to do a circleOperation with crappyApproximationOfPi = 3.14 (the product is cylindrical, and we need to know about how much ribbon we need to wrap the package when someone selects the gift wrapping option), who gets the naming rights?

If there is a line that should be drawn (and I'm not convinced there is), and there exists a place to put it, I don't think the DRY or YAGNI principles have anything to do with its placement. Readability is the key for me, and I don't know to what degree that can be quantified.

One in particular I dislike is when people have uncommented literals like 0x0032aBlahBlahBlah in the code. How (and why) on Earth would you expect anyone to know what that referred to? Will you remember the next time you look at it? I wouldn't, but my brain may just be a sieve.

Your thoughts on where to draw the line? (Or not draw it?... Or tell me I'm wrong?)

I can honestly say I've never had to build an application that needed to be portable between databases, aside from between different versions by the same vendor (although there was once where it might have been nice, since I used the apparently impossible to find in hosting combination of Microsoft SQL Server and Java). My guess is that unless you're building software which is meant to be deployed on different databases (such as a framework, library, or software you plan to sell over and over again which you won't be hosting), you probably haven't needed to do so either... Or at least, not often enough to let it impact your coding.

So today, I started my quest to turn cfrails into a DB-independent framework (as of today, March 30, 2007 it works only with Microsoft SQL Server). Since one of my goals is to limit the amount of configuration the programmer has to provide to get up and running, simply telling cfrails your database metadata (instead of having it look in the database itself) was out of the question (XML or otherwise). I've looked at using CF's getMetaData(query_result) function in the past, but that doesn't have near the amount of data that would be useful for me.

I figured my best bet would be to drop down into Java and use its DatabaseMetaData interface. This way, I could let Java take care of the differences between DB implementations for me. Sure enough, looking over the docs, it had everything I needed. I went ahead and whipped up some code to use it and test it before trying to put it into Coldfusion. If you're interested, here's most of what I needed (only using the jdbc/odbc bridge). If not, feel free to skip this code:


(if you need more info, see the docs I linked above -- this is just a small sampling of the metadata available for columns in a table)

Sweet! I didn't really need ORDINAL_POSITION since it already ordered the results by it, and COLUMN_DEF (the column's default value) wasn't all that important (well, I could figure out a way to get that later, anyway). But, when I went to put it into ColdFusion, I realized - I have no clue how I might get the information I need to connect- forget about the fact that I'm not sure the average developer (CF or otherwise) will know what a catalog or schema is, or which one they need to access if they do. Who in the world isn't going to have to go through some work to figure out what driver they need and what the connection URL is?

So, I needed a way to let Java know these things through CF. I thought I could get at least some of that information from the CFAdminAPI (or whatever the proper case and spelling is), but I didn't really like that option. I couldn't think of any other option that would allow me to use Java to get the metadata, so I decided I'd have a look at the codebases for Transfer ORM, Reactor, and DataMgr and see how those smart guys did it. I only browsed, but I didn't find anything useful for me trying to use Java to do the trick. So, it was back to writing a DB-specific implementation for each database I wanted to support.

Luckily for me, even though my design doesn't approach the modularity and complexity of Reactor or Transfer (yet... I'm a big fan of YAGNI and haven't had any reason to break out of the simple design I currently have and Transfer is downright confusing for me to quickly browse it and figure out what's going on), it is quite flexible enough for me to do the different implementations and integrate them with ease. To do so, I only need to refactor the current routine that gets the MSSQL Server metadata, and provide a way to figure out which database type it is looking in (each of these only has to happen once). After that, all that remains is to write a query to get the metadata and convert its output (for each implementation) to the interface the rest of my code expects. Not hard by any means, but it would have been cool for maintenance purposes and greater overall simplicity if I could have let Java handle it.

So now that you've gotten this far in this boring post with no conclusion, have you used Java to get the DB metadata in CF (or even outside of CF)? Have you found a way to do it without making the programmer specify the connection URL and driver?

At that point, I decided to go ahead and start coding. Now, I wasn't using TDD, since I have no clue how I might go about doing so other than to first code in different classes, and then migrate everything over to main(). In that case, I figured there may very well be more bugs after migrating, rather than less. So, I just started coding.

Once I got down to having to hit the database table to verify it and the columns existed, I got stuck. I didn't know if I needed to test against an actual DBMS using JDBC or if I needed to check against a .CSV file, for example. I stayed stuck and thinking for a while, until I started thinking in objects (or more generally, and importantly, abstract data types). At that point I realized that whatever I did, I'd have to store it some how, and in doing so, I could "fake it" and implement that part later. All I needed was the interface, essentially.

Now, this is a technique you'll often use in OOP and especially if you're following test driven development, but given the nature of the constraints, I didn't think about it immediately.

The point is that even if you aren't programming with objects or using TDD, you can still think about abstraction, and use it to continue coding even when the detailed implementation-level requirements aren't fully specified. I'm not sure this is something I would have thought about when programming purely procedurally, so hopefully it will help someone out, as it certainly did for me.

Thoughts anybody?

So, now we have a table, let's call it drink. It has columns id (int, pk, identity), brandname (nvarchar 50), label (nvarchar 50), bottle_date (datetime), and description (ntext). The code currently looks like:


In this case, assuming you have the corresponding controller and view set up, cfrails will show label as "Label" and bottle_date as "Bottle Date" to the user. description will be capitalized as well. Label will be shown as an <input type="text"/> and description as a <textarea>.

But while brandname would show up as an input text field too, it will be displayed as "Brandname." Now, if you had called the column brand_name or brandName, the space would be put in automatically. That's easy enough to do with new tables, but if you're working on an old one where you had used the convention numBottles, for instance, it would show up as "Num Bottles," which is not something you want displayed to the user.

Another issue shows up with bottle_date. Since it is a datetime column, cfrails will display it with a select for each of year, month, day, hour, minute (when I get around to implementing that). But in this case, you might only care about the month and the year.

How do you solve those two problems? That's where some of the configurations / extended metadata comes in.

One of the goals I have in cfrails is to keep related data together. I'm not fond of the thought that I might need to have to open an extra file to configure my class. So, if I want the view to show the name of brandname as "Brand Name," I should be able to do that within drink_view.cfc (actually, you can hand-code your views in templates if the customization options don't fit the bill, but I'm not getting into that here either).

What I wanted to be able to do was have my files look like this:


(As a side note, you could use the underscore version of those functions as well. Also, the month/YearDate has not yet been implemented as of Feb. 20, 2007, and is subject to change, but you get the idea.)

That certainly seems easy enough. But, since the controller has to pass the model into the view through the view's init() method, that configuration code at the top will be executed before the view has any knowledge of a field called "brandname."

It was a long time coming, but we've finally reached the point where I needed a stack (well, the order didn't really matter, so what I implemented wasn't really a stack, but it served the same purpose).

Instead of running the actual function when it is called, I checked if the component was fully instantiated. If not, I "stack" each function that needs to be run, along with the supplied parameters. Then, in the init() method, the stacks are "popped" (it's actually done in a queue fashion, but it feels odd saying that).

Thoughts?

I think this fits quite nicely with a link Peter Bell put on his blog not that long ago, Programming by Wishful Thinking. Programming by wishful thinking would be at a higher level, yet lower (as I understand it), but it looks like it would dovetail nicely with the PPP. Its a bit off topic, but to explain why I say higher level, yet lower, I just mean that in PBWT, you'd be writing statements in your language of choice to implement a solution, and then you'd go implement those methods (this also works well in TDD. So at the level of the statement, you're writing code, so it is lower level, but it is higher level in that you are designing a higher level routine that would use these helpers. You'd likely use the PPP in both "levels" if you will, though it would probably be more helpful in the lower ones, since the PBWT should be very self-explanatory.

I used to take this approach (described in the PPP), but stopped for reasons I'm unsure of. I liked it when I did it, but I guess I got out of the habit. But, you can be sure I'll be trying it out again, especially since a lot of the stuff I'm working on lately is really dynamic programming, with little "real-world" objects to think about.

One long quote which prompted this post that I wanted to share comes when McConnell is describing that as part of the process, you will be naming your method. Of course, we know that a method name should be unambiguous, because it might signify a problem (lack of cohesion, in particular) if you cannot name it without the ambiguity. Here's what McConnell had to say:

Naming the routine might seem trivial, but good routine names are one sign of a superior program and they're not easy to come up with. In general, a routine should have a clear, unambiguous name. If you have trouble creating a good name, that usually indicates that the purpose of the routine isn't clear. A vague, wishy-washy name is like a politician on the campaign trail. It sounds as if it's saying something, but when you take a hard look, you can't figure out what it means. If you can make the name clearer, do so. If the wishy-washy name results from a wishy-washy design, pay attention to the warning sign. Back up and improve the design.

Sound advice, no?

PS: I finally upgraded to Firefox 2.0 (I don't know what took me so long, but apparently I was still running 1.0 on this computer) and I'm loving that spell-check feature.

Both points are things most of us already know, but things you sometimes lose sight of when working like that all the time, or without paying attention. But, I was paying attention on Monday, so let me explain how I came to identify those points.

We started by trying to identify a class we could test. This was harder than it looked. We initially decided on Game. There was a lot of discussion on making it an abstract class, or an interface, or should we just make it support only tic-tac-toe? After everyone was convinced that we should first make it concrete, because we would not be able to test it otherwise, we started trying to figure out what we could test. So, we wrote a test_hookup() method to check that JUnit was in place and working.

In the end, we also decided that we would probably want to try to make it abstract after we tested the functionality, so we could support multiple games. I think that decision proved to be a bad one - because none of us could figure out a single peice of functionality that a Game should have. For me, I think this was due mostly to the fact that I was trying to think of general functionality, so I only came up with run(), basically the equivalent of play().

After thinking for a couple of minutes about things we could test, we decided we should go with something more tangible for us - the Board class. Once we did that, we never had a problem with what to test next. But, we still ran into problems with YAGNI. We were arguing about what type of object we should use, so I just said "let's use Object, then anything will fit and we can be more general when we get back to Game." This led to trouble later, when we had to have checks for null all the time. So, we changed it to a string whose value could be "x", "o", or "". That made life, and the code, a lot simpler.

Those are the two main places where ignoring the YAGNI prinicple hurt us. There are also a couple of places where not writing our tests first hurt us.

The most obvious one is that we were just looking for a green bar. Because of that, we actually were running the wrong tests, and some of ours hadn't worked in the first place. We were running the test from the Game class, not the Board. It wasn't too hard to fix the ones that were broken though.

Other than that, we spent a lot of time trying to figure out if an algorithm to determine a winner would work (in our heads). If we had just written the test first, we could have run it and seen for sure if the algorithm was going to work correctly.
Overall, the only other thing I would have changed about how we developed would have been to decide on several tests in advance, so we would have plenty to choose from, rather than thinking of a test, then immediately implementing it.

For reference, I've posted our code below.


As you can see, the board object really didn't do what its name implies. That should have been game, I think. Also, the three check for winner functions are only temporarily public - we planned on combining them to check for a winner in just one public function. Other than that, we also need some negative tests, and obviously to complete the game.

Finally, thanks to everyone who showed up for making it an enjoyable experience! I can't wait for the next one!

But sometimes, I think I can go further. And sometimes I do, while others I don't. There is a common thread between most of these times I feel I could go further: if I could name the block of code. And the commonality between most of the times I don't extract that block into a method: the code is not especially complex, and I am too lazy to do it "right now."

So, I wonder, would it be a fairly valid rule to say "if you can name a block of code, put it in a method"? Of course, you can't simply say "if you can't name it, don't put it in a method," since it might be hard to think of a good name (or, it may just be that the code is not cohesive). But I think the other way might go a long way to reducing complexity and making the code more easily understood.

What do you think? Do you follow any guidelines that you've noticed (either intentionally or not)?

I'm not convinced there is some arbitrary line where abstraction would increase complexity. In fact, as it is designed to reduce it, I'm not sure you'd be doing it right if it did increase complexity.

I would also point out that besides the abstractions we create in Coldfusion, you might very well see Coldfusion itself as an abstraction on top of Java, which in turn is an abstraction of C/C++ (No more pointers!), which in turn is an abstraction over assembler, which is an abstraction over machine code... and so on and on it goes.

Still further, we might want to abstract the idea of a queue (for instance) from handling it as an array (or your preferred implementation). But that doesn't mean we wouldn't further abstract it as something else to keep our domain consistent.

In any case, even though, yes you are adding code to do something you can do without code in the case of a generation script, I see that as a valid approach to reducing complexity, especially if you find XML to be generally more complex than scripts. Because once you've abstracted it, you no longer have to think about it.

And so we have something to test with, I've made this simple test.cfc:


And finally, here we see how to create a pseudo-object of test.cfc:


Now, it was fun to play around with this stuff, but I realize it's probably useless (I mean, I didn't do any benchmark testing to see if it was faster than object creation, since I had to read and write to the file system). Further, I'm not at all happy with the design. My main concern is that the semantics are exposed: a user of this code needs to know they have to call the methods in the order shown in test_pseudo_object.cfm to have it work. That could be fixed however, if I required that the name of one of the variables (and its scope) be passed into the CFC via the init() method. Then, it would return the struct, all in one go. In fact, if I were to ever start using this, I would reimplement it in that fashion, and it wouldn't be very hard.

Anyway, what do you think of this useless ...

I think part of it may be due to bad habits and not having the ability to notice a spot where I need another class. But, I have no evidence of this, since I haven't been able to notice it =). I do have evidence of the other two options however: CF being so much more high-level than Java is fairly self-evident: Where I'll often use a façade to interact with a database in Java, in Coldfusion it would be pretty useless - since there is no need to instantiate a connection, prepare a statement, prepare a record set, and then close/destroy them all when you don't need them any more. The other, about the performance hits, I know sometimes stops me from creating more classes. For instance, I recently used a mixin to achieve some code reuse with a utility method that didn't belong to any of the classes that used it. Clearly it broke the "is-a" relationship rule, but I didn't want to incur the cost of instantiating an object just to use a single, simple method (the application was already seeming to perform slow).

Of course, this is not a zero-sum game. It is certainly due in part to all three. I'd like to think it is mostly due to the expressiveness of Coldfusion, but I'll always have to wonder about the bad habits.

In Ruby, I've developed a couple of "significant" applications, but those didn't approach the size of the one's I've done in Coldfusion or Java. It's hard to say for sure, but I'd guess that I'm writing much fewer classes in Ruby than I would in Java. Again, this is mostly due to Ruby's expressiveness.

In any case, have you noticed yourself writing fewer classes in CF than in other languages? Have you thought about it? Come to any conclusions?

In a nutshell, the goal is to have such highly customizable "scaffolding" that there really is no scaffolding - all the code is synthesized - or generated "on the fly." Of course, this only gets you so far. For the stuff that really makes your application unique, you'll still have to code that. But you can compose views from others and such, so it's not like related tables have to stay unrelated, but I do want to stress that right now there is no relationship stuff implemented in the ORM.

I've skipped a few "mini-versions" from 0.1.3 to 0.2.0 because there were so many changes that I haven't documented one-by-one. That's just sloppiness on my part. Basically, I started by following Ruby on Rails' example, and taking my own experience about what I find myself doing over and over again. That part is done, except that the ORM needs to be able to auto-load and lazy-load relationships at the programmer's whim. In any case, once I got enough functionality to start using it on my project, I've been developing them in parallel. The problem is, I've fallen back on poor practices, so the code isn't as nice as it could be.

In particular, there aren't any new automated tests after the first couple of releases, which isn't as bad as it might otherwise be, since a lot of the core code was tested in them. But on that note, I haven't run the existing tests in a while, so they may be broken. Further, since I've been thinking in Ruby and coding in Coldfusion, you'll see a mix of camelCase and under_score notations. My original goal was to provide both for all the public methods, and I still plan to do that (because, since I can't rely on case from all databases for the column names -- or so I think -- I use the under_score notation to tell where to put spaces when displaying the column names). But right now, there is a mix. Finally, the DSL needs a lot more thought put behind it - Right now it is a mix-and-match of variables.setting variables and set_something() methods. Right now it is really ugly, but when I take the time to get some updated documentation up and actually package it as a zip, I should have it cleaned up. In fact, I shouldn't have released this yet, but I was just starting to feel I needed to do something, since so much had be done on it and I hadn't put anything out in quite some time. Besides that, I'm quite excited to be using it - it's been a pain to build, but it's already saved me more time than had I not done anything like it.

In the end, I guess what I'm trying to say is: 1) Don't look at it to learn from. There may be some good points, but there are bad points too, and 2) Don't rely too heavily on the interfaces. While I don't anticipate changing them (only adding to them, and not forcing you to set variables.property), this is still less than version 1, so I reserve the right to change the interfaces until then. =)

Other than that, I would love to hear any feedback if you happen to be using it, or need help because the documentation is out of date, or if you tried to use it but couldn't get it to work. You can contact me here. You can find cfrails at http://cfrails.riaforge.org/ .

His each() method is only slightly different from mine, but significantly so:


He mentions that I "[name] the method and then repeatedly [bind] the value variable. That doesn't actually do what he thinks." He's right. I was using it incorrectly, and I thought that it was sort of odd to do it like that. In particular, I didn't like having to name the closure. This syntax is much cleaner, and easier to understand. And the best part follows: I had mentioned that, "from my understanding of closures, you should be able to modify 'outer' variables within them. Thus, to be a 'true' closure, outputting beenhere above should show true" in reference to this code:


Sean shows that in fact, had I used bind() as it was intended, outputting beenhere would have shown true, as expected. Basically, you can use bind(outer=variables) on the closure to get access to it. Then, in the code for the closure, you could use outer.beenhere = true and of course, the value is changed as you would expect. That is close to what I was thinking of when I said my idea would muck up the syntax, but I was thinking of passing it when creating the object. I actually like this a little better, as it is much cleaner than what I envisioned, and it is easier to see what's going on. It would be nice of course, if there was a way to do it without the programmer worrying about it, but I don't think that is possible with CF, since it uses the the same name for (from what I can tell) are two different scopes.

Anyway, thanks again to Sean for cleaning up my mess.

Luckily for us, Sean Corfield made a Closures for CF package available a couple of months ago (as far as I can tell). A couple of days ago, I gave it a whirl.

The first thing I have to say is that I had to move the CFCs into the same directory as my closure_test.cfm file because it wasn't able to find the temporary templates it was generating. Sean let me know it was because of a pre CF 7.0 bug in expandPath(), which explains why I had trouble with that function in cfrails. Anyway, I've posted a "fix" for 6.0/6.1 to the comments on his blog, which basically just replaces expandPath(".") when it is creating a temp file to use getDirectoryFromPath(getCurrentTemplatePath()).

After that, it did take a while to understand what was going on, but once I figured it out, it was pretty cool. So here's my first class of using closures, and what I did in Coldfusion to implement it: really "encapsulating" something.

Encapsulation is a word you see thrown around quite often. But you don't achieve it by putting getters and setters everywhere - particularly if you are passing back complex data structures from a get() method (without first making duplicates, of course). The idea behind it, as far as I've been able to understand, is to hide implementation details from code which doesn't need to know it. Not only does this make the client code need to be less complex, it certainly lowers your coupling as well. However, if you have a Container (for instance), how can you encapsulate it? If you return an array representation, you've just let the client code know too much. If you don't let the client code know too much, how will it ever iterate over the contents of your Container?

Certainly an Iterator comes to mind. You could provide that if you wanted to, complete with next(), previous() and anything else you might need. But you could also use a closure to allow you to encapsulate iteration over the Container, while still allowing you the freedom to do what you wanted within the loop. Here's my CF code:


And here's the resulting code that uses Container.each(), and passes a closure to it:


Now, from my understanding of closures, you should be able to modify "outer" variables within them. Thus, to be a "true" closure, outputting beenhere above should show true. This would be relatively easy to do by sending the appropriate scopes to the closure, but the only ways I can think of to do that would muck up the syntax. Other than that, it is not code that makes me comfortable, so to speak. The closure_test needs to know that it is expected to use value as the name of the variable. There may very well be a simple way around this that doesn't screw up the syntax, but I have yet to spend enough research in figuring it out. In fact, in Ruby you'd use |value| do_something_with(value), so it's probably as simple as that, or providing an extra parameter (as I saw in one of Sean's examples). Perhaps I will when I run across a need for it (and I'm sure I will be using this in the future).

And now for comparison purposes, I've included the Ruby version that does the same thing:


This isn't normally how you'd code in Ruby, but I tried to keep the look approximately the same. In particular, the each() method is already implemented for arrays in Ruby, so there would certainly be no need to rewrite it.

As you can see, the syntax is a bit cleaner in Ruby than in the CF version. It's so easy to do, one doesn't need to think about using closures - you just do it when it comes naturally. Even with all that, I think Sean reached his goal of proving himself wrong (he said "I was having a discussion about closures with someone recently and I said they wouldn't make sense for ColdFusion because the syntax would be too ugly. On Friday evening, I decided to try to prove myself wrong"). I think it might be possible to improve it a little, but I don't think it would ever become as "pretty" as Ruby's style (unless, of course, it was built into the language). Because of that, and the effort involved in creating them, I don't think it will ever become a natural solution we turn to in CF (although, if I started using them enough, it would eventually become a natural solution). In any case, he certainly did an awesome job, made the syntax much cleaner than I would have thought possible, and gave us a new tool to solve problems with.

In closing, I'd like to talk a bit about the "other" use for closures I've found, what I called a generalized Template Method pattern above. Really, this is just a general case of the way encapsulation was furthered with the each method above. Basically, you are defining the skeleton of an algorithm, and allowing someone else to define its behavior. I mentioned that it's not really the template method pattern, since we aren't confining it to subclasses, and we aren't actually following the pattern - we're just implementing its intent, in a more general way. So basically, you might think of it as a delayed evaluate for letting some other code customize the behavior of your code that expects a closure. I don't know if I've made any sense, but I sure hope so (and if not, let me know what I might clarify!).

Finally I'd like to thank Sean for providing this tool to us, and ask everyone, "what uses have you found for closures, if any?"

Update: Thanks to Brian Rinaldi, Sean Corfield found this post and addressed the problems I encountered. It turns out I was just doing it wrong, so to see it done right, visit his blog. I also responded to a few of the issues raised here.

In Java/JSP, no, I would almost never write the HTML in a class. I may have some small bits in a UI class from time to time, but never more than a single tag or perhaps two. But, it is not for any other reason than writing HTML like the following is completely unreadable and impossible to follow, and more importantly easy to screw up:


Can you imagine doing an entire page like that? Or even just a table with more than a couple of columns? It's unwieldy.

With CFC's however, you can still keep the HTML in its readable form. In some cases, I prefer to use CFCs to accomplish the goal of reuse, while in others, I prefer to use includes. I find myself preferring CFCs in particular, in cases where there may be slight customizations to a basic display, based on individual client preferences. I can inherit (or aggregate, if it makes more sense) from the base display (cut up into logical chunks) and reuse those parts which make sense, and only rewrite the custom bits. It lets me keep duplicate code to the absolute minimum - something I'm very fond of these days. The inheritance in particular has the extra benefit of allowing me to keep client code unchanged, as long as I've followed Liskov Substitution Principle.