Later, he moves on to describe four different types of design for your DSL. This was something I hadn't really thought about before, therefore it was the most interesting for me. Here are the four types:

• Instantiation: The DSL consists simply of methods on an object. This is not much different from normal programming. He says it is "DSLish," perhaps the way Blaine Buxton looks at it.
  
  
• Class Macros: DSL as methods on some ancestor class, and subclasses can then use those methods to tweak the behavior of themselves and their subclasses. This follows a declarative style of programming, and is the type of DSL followed by Ruby on Rails (and cfrails) if I understand him correctly.
  
  
• Top-level methods: Your application defines the DSL as a set of top-level methods, and then invokes load with the path to your DSL script. When those methods are called in the configuration file, they modify some central (typically global) data, which your application uses to determine how it should execute. This is like scripting, and (again) if I understood correctly, this is the style my (almost working correctly) Partial Order Planner uses.
  
  
• Sandboxing (aka Contexts): Similar to the Instantiation style, but with more magic. Your DSL is defined as methods of some object, but that object is really just a "sandbox." Interaction with the object's methods modifies some state in the sandbox, which is then queried by the application. This is useful for processing user-maintained scripts kept in a database, and you can vary the behavior by changing the execution context.

Note: These are "almost" quotes from the slides/talk (with a couple of comments by myself), but I didn't want to keep rewinding and such, so they aren't exact. Therefore, I'm giving him credit for the good stuff, but I stake a claim on any mistakes made.

The talk uses Ruby as the implementation language for the example internal DSLs, but I think the four types mentioned above can be generalized. At the least, they are possible in ColdFusion - but with Java, I'm not so sure. In particular, I think you'd need mixin ability for the Top-level methods style of DSL.

Next, Obie catalogs some of the features of Ruby you see used quite often in crafting DSLs: Symbols (because they have less noise than strings), Blocks (for delayed evaluation of code), Modules (for cleaner separation of code), Splats (for parameter arrays), the different types of eval (for dynamic evaluation), and define_method and alias_method.

Finally, he winds up the presentation with a bit about BNL. I liked this part because I found Jay Fields' blog and his series of articles about BNL.

As always, comments, questions, and thoughts are appreciated. Flames - not so much.

• Added the automatic include of views/layouts/application_header.cfm and views/layouts/application_footer.cfm even when using view CFC (formerly, this only happened if you instructed it to and using .cfm templates for views). Also added auto-include of controller-specific templates simply by placing controllerName_header.cfm and controllerName_footer.cfm in views/layouts.
  
  
• Brought the existing unit tests up to date (amazingly, with all the neglect there was only a couple of very minor problems, mostly due to changing paths).
  
  
• Migrated request processing out of individual applications and into Dispatcher.cfc. This is a major improvement in the DRY arena, and allows me to make changes to the ways requests are processed without having to make them to every application that uses cfrails (Thanks to Dan Lancelot for prodding me to do this a couple of months back).
  
  
• Added some minor Spry integration for form validation - this only works with required text fields at the moment.
  
  
• Added the ability to use primary keys that aren't named "id" (although, still no composite PK support)
  
  
• Changed behavior of a particular file to not catch errors (it was getting hard to debug)
  
  
• Added an onchange attribute for form elements (can edit onchange per column without writing out the entire form)
  
  
• cfrails now automatically creates blank controller, model, and view cfcs if you request one where it doesn't exist. This just makes it easier to develop, and may end up being a setting you turn on and off.
  
  
• Added validations where you can specify a function to call and a message to display upon failure for validating forms (as opposed to simply using the built-in autovalidation based on DBMS metadata).
  
  
• Added support for bigint, tinyint, and smalldatetime data types.
  
  
• Added the ability to call a function when outputting a column in a list (though complete closures are not yet fully integrated)

So, there was a lot done as you can tell. And, MySQL support is coming soon, I promise!

Update: Oh yeah, I forgot what might be the most important part: New, more complete docs on cfrails are available (thanks to cfcdoc).

Of course not -- some fields should map to a foreign key which should show up as a select form element, which lists the human readable descriptions of those fields (as opposed to their database identifiers). Further, some fields should be hidden, and some fields should be compositions of others. Values for some fields should be in a range of legitimate values, and so the list of possible customizations goes on. In those special cases (which occur frequently enough), you just need to provide a means to override the default behaviors (through extra, user-defined metadata, a DSL, or some other tactic or a combination of them).

Then, in the really rare edge cases, you can always drop out of the framework and code it yourself (or have it generate a template for you to modify). Even for search forms (which I think are more often in need of customization than simple CRUD stuff), you can provide the basics and allow the programmer to customize as needed (and drop out of the framework all together if need be).

But is there a similar strategy for generating reports? My initial thought is that there isn't. Reports just seem to need too much customization to have any useful automagic generation. But, I'm trying to look past that initial impression, because I don't have as many years of experience in hand-writing the same crap code over and over again in reporting as I do in writing CRUD operations (now, I have the same crap code for CRUD in one place and it just figures out what to do, but at least I don't ever need to write it again =)).

In any case, I just don't see it. But, I can see using metadata/DSL, where you can define the tables and columns you want to report on, and generally it will still be much faster for developing reports (can you tell I'm trying to work on this at the moment?). Taking the idea further, what about bringing convention over configuration into this realm? Once you get out of the world of 8.3 filenames and realize you can name variables (or columns) pretty much anything you want, it is easy to see the benefit of having names that describe what you want the user to see. For instance, instead of orderGTot, just go ahead and name the column order_grand_total or orderGrandTotal. Who cares about the extra characters? It's not like you have to type them that many times, since the framework (presumably) takes care of most of it for you, and you don't have to define what the user should see in more than one place.

Of course, this is also just metadata. But, instead of having what amounts to the same metadata in different places, you've just got it in one - your column name. So if we take this idea to reporting, what if we just tag certain columns with something like _reportable? I haven't figured out how we might generate reports on that amount of information alone, but I see it as a useful start.

When your classes average between zero and twenty lines of code (which amounts to just customizing interfaces and performing validations, along with perhaps a couple of calculations), it's time to stop worrying about DAOs, DTOs, PPOs, and Cheerios. Skip to the IPO and let the frameworks worry about the rest. I want one to do my reporting for me, because that seems to be one of the last major hurdles that I see (at least, as far as getting rid of repetition goes).

What do you think?

My interest lies in the second complaint though - that writing it to a live file (rather than dynamically figuring it out - which I've been calling "synthesis," as opposed to generation) means you can't get the benefits of upgrades to the scaffolding engine (and, it is not following DRY!). But, if you just use the default scaffolding, what happens if it doesn't come out just right (assuming you've even passed the notUgly test in the first place)? Well, thats a big part of what I'm trying to solve with cfrails, by using a DSL that provides tweakability to the scaffolding, without requiring you to write to a file (though, if the tweaks won't work, you are always welcome to go to a file with HTML and the like). The interesting part for me about the RoR plugins above, therefore, is that it appears (I haven't checked them out yet) that at least Hobo contains a DSL, called DRYML, to help along those lines.

I'll be having a closer look at those when free time becomes a bit less scarce. What do you think? Is it the holy grail, or can there be very useful "scaffolding"?

Being that I've been burned a lot in the past by not following the DRY and OAOO principles, I'm quite fanatacal about following them nowadays. So when I needed the list() function to behave a bit differently for this one type of object, I didn't want to replicate the entire list function -- I just wanted to add a bit of logic in the middle. That's where closures enter the picture.

I decided I'd create a function, onOutputOf(column="column that special stuff needs to be done to", run="function, or code"). So, instead of outputting the value of the cell, the list now knows to run the code provided in the run parameter.

And now, some really complex code doesn't need to be replicated elsewhere. Isn't that sweet?

Then, of course there are the generic getters and setters that Peter Bell is often talking about. You know, where you have some implementation like (excuse the pseudocode -- I'm just lazy right now =), but it shows the idea):


This is easy enough to implement without resorting to file I/O, and it has the side benefit of allowing you to check if a getAttribute() method already exists, and call it if so. And where this morning starts getting wierd is that I randomly came across this post from David Harris where he and Peter are discussing this strategy in the comments. What a coincidence.

But what I really wanted is something you see in Ruby (and other languages too): the attr_accessor (or reader and writer) in Ruby. You can do something like this:


See how easy that is?! That's what I really want to be able to do in ColdFusion. And the best part is, it really would only (as far as I can tell) require a couple of changes to the language. The first one being something like if (cowtype= is defined as a method on the object I'm working with) call it when doing an assignment; and the second one is being able to define a method like so:


In any case, the morning got more coincidental when I ran into this post on InfoQ about Adding Properties to Ruby Metaprogramatically, which I'd recommend reading if you're wanting to metaprogram with Ruby (or if you are just interested in that sort of thing).

It's now time for me to crawl back into my hole and write about the wonderful world of The History of Partial-Order Planners. (The good news on that is I'm getting close to the fun part - actually programming one).

So method= is going on my wishlist for CF9, and in the mean time I'll probably end up going the generic getter/setter route. What are your thoughts? What would you go with?

I can honestly say I've never had to build an application that needed to be portable between databases, aside from between different versions by the same vendor (although there was once where it might have been nice, since I used the apparently impossible to find in hosting combination of Microsoft SQL Server and Java). My guess is that unless you're building software which is meant to be deployed on different databases (such as a framework, library, or software you plan to sell over and over again which you won't be hosting), you probably haven't needed to do so either... Or at least, not often enough to let it impact your coding.

So today, I started my quest to turn cfrails into a DB-independent framework (as of today, March 30, 2007 it works only with Microsoft SQL Server). Since one of my goals is to limit the amount of configuration the programmer has to provide to get up and running, simply telling cfrails your database metadata (instead of having it look in the database itself) was out of the question (XML or otherwise). I've looked at using CF's getMetaData(query_result) function in the past, but that doesn't have near the amount of data that would be useful for me.

I figured my best bet would be to drop down into Java and use its DatabaseMetaData interface. This way, I could let Java take care of the differences between DB implementations for me. Sure enough, looking over the docs, it had everything I needed. I went ahead and whipped up some code to use it and test it before trying to put it into Coldfusion. If you're interested, here's most of what I needed (only using the jdbc/odbc bridge). If not, feel free to skip this code:


(if you need more info, see the docs I linked above -- this is just a small sampling of the metadata available for columns in a table)

Sweet! I didn't really need ORDINAL_POSITION since it already ordered the results by it, and COLUMN_DEF (the column's default value) wasn't all that important (well, I could figure out a way to get that later, anyway). But, when I went to put it into ColdFusion, I realized - I have no clue how I might get the information I need to connect- forget about the fact that I'm not sure the average developer (CF or otherwise) will know what a catalog or schema is, or which one they need to access if they do. Who in the world isn't going to have to go through some work to figure out what driver they need and what the connection URL is?

So, I needed a way to let Java know these things through CF. I thought I could get at least some of that information from the CFAdminAPI (or whatever the proper case and spelling is), but I didn't really like that option. I couldn't think of any other option that would allow me to use Java to get the metadata, so I decided I'd have a look at the codebases for Transfer ORM, Reactor, and DataMgr and see how those smart guys did it. I only browsed, but I didn't find anything useful for me trying to use Java to do the trick. So, it was back to writing a DB-specific implementation for each database I wanted to support.

Luckily for me, even though my design doesn't approach the modularity and complexity of Reactor or Transfer (yet... I'm a big fan of YAGNI and haven't had any reason to break out of the simple design I currently have and Transfer is downright confusing for me to quickly browse it and figure out what's going on), it is quite flexible enough for me to do the different implementations and integrate them with ease. To do so, I only need to refactor the current routine that gets the MSSQL Server metadata, and provide a way to figure out which database type it is looking in (each of these only has to happen once). After that, all that remains is to write a query to get the metadata and convert its output (for each implementation) to the interface the rest of my code expects. Not hard by any means, but it would have been cool for maintenance purposes and greater overall simplicity if I could have let Java handle it.

So now that you've gotten this far in this boring post with no conclusion, have you used Java to get the DB metadata in CF (or even outside of CF)? Have you found a way to do it without making the programmer specify the connection URL and driver?

I'm a big fan of choice, and of learning, so I tend to think that having more choice (to learn from as well) is better, but if choices are substantially similar, one should certainly strongly consider contributing to the similar project. Of course, a question reveals itself here: "how can I be sure what I'm doing is substantially different?" One cannot take the time to review every framework out there, and probably asking for just a couple is too much. In my case, I didn't review all the frameworks I knew about, but I did read up on them, at least. (As a side note, I don't know when Model Glue started generating code, can anyone confirm when for sure - has it always been there?)

I feel like cfrails is "substantially" different than anything I've seen (even CF on Wheels, which I also plan to contribute to), but I always worry if what I'm doing may at some time be considered to be stepping on toes. I don't want to be perceived that way, for sure. Further, I don't think I'm solving any problem that hasn't been solved before, but I do think I'm doing it in a different way - and I think there is merit in that, even if, in doing so, I am duplicating some of the effort of others (for instance, cfrails has its own ORM, but I felt that changing what I knew about the existing ones might make them more specific than they were intended to be). With all that being said, I think generating code, while certainly being nothing new in computer science or even ColdFusion, is relatively new to being done publicly or in Open Source in ColdFusion. So, even though there are a lot to choose from, right now, it is a good thing, in my opinion, as we are all able to learn from each other, and may discover better ways of doing things.

In the end, as far as the generators go, it seems most of them are quite different in how they tackle the problem, and some of them even in the problems they solve. I think it would be great if those behind the various frameworks would write about how they came about (or, if you don't mind, leave a link here if you've already written it) and what problems they are intended to solve, and how they solve those problems. That might be a great learning exercise for all of us in the community (I plan to write that up for cfrails after the next couple of weeks -- midterms, you know? =) ).

So, now we have a table, let's call it drink. It has columns id (int, pk, identity), brandname (nvarchar 50), label (nvarchar 50), bottle_date (datetime), and description (ntext). The code currently looks like:


In this case, assuming you have the corresponding controller and view set up, cfrails will show label as "Label" and bottle_date as "Bottle Date" to the user. description will be capitalized as well. Label will be shown as an <input type="text"/> and description as a <textarea>.

But while brandname would show up as an input text field too, it will be displayed as "Brandname." Now, if you had called the column brand_name or brandName, the space would be put in automatically. That's easy enough to do with new tables, but if you're working on an old one where you had used the convention numBottles, for instance, it would show up as "Num Bottles," which is not something you want displayed to the user.

Another issue shows up with bottle_date. Since it is a datetime column, cfrails will display it with a select for each of year, month, day, hour, minute (when I get around to implementing that). But in this case, you might only care about the month and the year.

How do you solve those two problems? That's where some of the configurations / extended metadata comes in.

One of the goals I have in cfrails is to keep related data together. I'm not fond of the thought that I might need to have to open an extra file to configure my class. So, if I want the view to show the name of brandname as "Brand Name," I should be able to do that within drink_view.cfc (actually, you can hand-code your views in templates if the customization options don't fit the bill, but I'm not getting into that here either).

What I wanted to be able to do was have my files look like this:


(As a side note, you could use the underscore version of those functions as well. Also, the month/YearDate has not yet been implemented as of Feb. 20, 2007, and is subject to change, but you get the idea.)

That certainly seems easy enough. But, since the controller has to pass the model into the view through the view's init() method, that configuration code at the top will be executed before the view has any knowledge of a field called "brandname."

It was a long time coming, but we've finally reached the point where I needed a stack (well, the order didn't really matter, so what I implemented wasn't really a stack, but it served the same purpose).

Instead of running the actual function when it is called, I checked if the component was fully instantiated. If not, I "stack" each function that needs to be run, along with the supplied parameters. Then, in the init() method, the stacks are "popped" (it's actually done in a queue fashion, but it feels odd saying that).

Thoughts?

I also noticed that I needed to mix views on occasion (for instance, a Person_view might need to display a list of checks written, which would be in another component). For this reason, I added getModel(modelName) and getView(viewName) to the mix. This is a great abstraction to remove the long and repetitive createObject calls. The downside is that apparently, CF6.1 and CF7.0 look components up differently, so these don't work (depending on your directory structure, I think) in the older version of CF.

The most significant addition, however, is that there is now some ability for relationships among tables to be added into your models without much effort: via a call to the autoload_relationship() method in your models. This was a bigger pain than I thought it would be, but well worth the effort.

Overall, I'm starting to get excited about the progress being made on this. I've been using it to extend an application we had written some time ago, and the results are good. For comparison, I took one of the "old" modules that was written without cfrails, of similar size to one I am working on now. The one written using cfrails is a bit smaller in scope and less complex, so I added in the other 2 "modules" I've been working on using it as well. Whereas the "old" one was over 500 lines, the 3 parts (two are quite small) using cfrails total to about 65, including white space. I spend more time marveling at the thing than I do programming (just kidding, of course =) ). Of course, this isn't any scientific comparison, especially since I'm sure I've grown as a programmer since writing the other one. But, it is still pretty remarkable. The hardest part is going to be memorizing the function calls. Well, not that it's hard, but it will take time looking at the (almost non-existent) docs, which may slow you down.

So, what's ahead? First I need to reconcile differences between CF7 and 6.1. After that, I need to add in abstractions to make "select" form inputs, as well as groups of radio buttons and checkboxes. There are some other abstractions I'm thinking of putting in, but I'm not sure if they add any value or just clutter the interface, so I'm leaving them out for now. Other than that, there are some more options I'd like to put in, like concatenating columns of a (HTML) table, to make the list more customizable. Finally, a few more major things are integration with an Ajax framework, integration with more DBMSs (right now, only SQL Server 2000 and 2005 are supported), and I'm toying with the idea of automatically loading referenced tables when there are columns of the type tablename_id. The biggest thing is re-writing the documentation. The only docs I have right now are based on version 0.1.0. Clearly, a lot has changed since then.

Those are the things off the top of my head. Even though the version number is only 0.2.1, I'm starting to feel awfully close to version 1. The main thing is that I want the interfaces (of the code variety) to be intuitive and easy to use, so I'm getting practice in using it. I may try to write a different application using it before I go to version 1, to make sure that it remains as easy to use in that one as it is in the application I'm using it for now.

But surely you can't provide every combination! That's where you'd want to provide a filter argument. Here it comes again: another but! You don't want to simply provide filter="where column1=#form.column1#" - you want to parameterize the query for the developer. Now, one way to do that would be to parse the filter argument and reconstruct it using your cfqueryparams in all the right places. But, an easier way would be to something like this:



Note: I would probably move the "where" into the abstraction, and just let the programmer provide the clause, but for illustration purposes, I left it in there.

For illustration purposes, my function only does this:


But it would be easy enough to modify it to do something useful. Right now, if you run it, you see all the arguments are available in the arguments structure. The first one comes under the key "filter", and the rest come under numeric keys, in the order they were passed (of course, looping over it won't show you that order, but the number of the key represents the position in which it was passed. Now, you can parameterize the arguments, first checking more restrictive and moving to less restrictive parameters (for instance, you'd want to check isDate(), isNumeric(), and fall back on string).

You could name the arguments too, but I haven't found that useful for this example, since they will have their names as keys in the arguments structure, and I know of no good way to figure out the order in which they were passed.

Finally, I haven't yet implemented this in cfrails, so I don't know how feasible basing parameterization on the CF type is, but I'll try to remember to post it when I do implement it. In any case, I'd be willing to bet there are other uses for this - it's just that this is the most important one on my mind recently.

Update: It did occur to me as I was writing this post that you could accomplish the same thing by using an array as a defined argument, with the added benefit of keeping your interface well defined. In all honesty, I would prefer to do it that way if you could define an array inline like: [9, "blockhead", "charlie brown"]. But the way you define arrays in Coldfusion really makes for too much code in what I'd like to accomplish with this, so in this case I'd provide a hint in the function to let a developer know it expects you to pass those arguments. This way, I get cleaner syntax and a decent solution to the interface problem.

In a nutshell, the goal is to have such highly customizable "scaffolding" that there really is no scaffolding - all the code is synthesized - or generated "on the fly." Of course, this only gets you so far. For the stuff that really makes your application unique, you'll still have to code that. But you can compose views from others and such, so it's not like related tables have to stay unrelated, but I do want to stress that right now there is no relationship stuff implemented in the ORM.

I've skipped a few "mini-versions" from 0.1.3 to 0.2.0 because there were so many changes that I haven't documented one-by-one. That's just sloppiness on my part. Basically, I started by following Ruby on Rails' example, and taking my own experience about what I find myself doing over and over again. That part is done, except that the ORM needs to be able to auto-load and lazy-load relationships at the programmer's whim. In any case, once I got enough functionality to start using it on my project, I've been developing them in parallel. The problem is, I've fallen back on poor practices, so the code isn't as nice as it could be.

In particular, there aren't any new automated tests after the first couple of releases, which isn't as bad as it might otherwise be, since a lot of the core code was tested in them. But on that note, I haven't run the existing tests in a while, so they may be broken. Further, since I've been thinking in Ruby and coding in Coldfusion, you'll see a mix of camelCase and under_score notations. My original goal was to provide both for all the public methods, and I still plan to do that (because, since I can't rely on case from all databases for the column names -- or so I think -- I use the under_score notation to tell where to put spaces when displaying the column names). But right now, there is a mix. Finally, the DSL needs a lot more thought put behind it - Right now it is a mix-and-match of variables.setting variables and set_something() methods. Right now it is really ugly, but when I take the time to get some updated documentation up and actually package it as a zip, I should have it cleaned up. In fact, I shouldn't have released this yet, but I was just starting to feel I needed to do something, since so much had be done on it and I hadn't put anything out in quite some time. Besides that, I'm quite excited to be using it - it's been a pain to build, but it's already saved me more time than had I not done anything like it.

In the end, I guess what I'm trying to say is: 1) Don't look at it to learn from. There may be some good points, but there are bad points too, and 2) Don't rely too heavily on the interfaces. While I don't anticipate changing them (only adding to them, and not forcing you to set variables.property), this is still less than version 1, so I reserve the right to change the interfaces until then. =)

Other than that, I would love to hear any feedback if you happen to be using it, or need help because the documentation is out of date, or if you tried to use it but couldn't get it to work. You can contact me here. You can find cfrails at http://cfrails.riaforge.org/ .

Since these applications are already well established, I didn't bother creating any layouts - so the page titles aren't exactly representative of each page, but it helped in that I could just wrap cfrails with the current template, and I was done.

Except, for one last issue: clicking on links back to the "old" part of the application meant that cfrails was looking for actions called "thispage.cfm," and didn't know how to handle it. So, in the newly (re)named cfrails.cfm, I just needed a simple check (after the action was extracted from the URL):


It's not something anyone couldn't have figured out, for sure. But, I was just amazed at how easy it was, because this was never a design consideration. I was building it for use on a completely fresh application.

Now, I've known this for quite some time - it's still the same box we were running CF 4.5 on. I think it may have had a couple of upgrades since then, but it's certainly not more than 528 MB RAM with a 1 GHz processor. In any case, I was doing some work this morning in a non-cfrails utilizing project, and it is equally slow (whereas a few months ago, when I was developing heavily on it, it was "normal"). That means it's likely cfrails performance isn't as bad as I thought it was, though I won't stick my neck out and say it performs well.

The further good news is that since I'm working in CF again, I'm wanting to use cfrails for the new work - and that means I'll be adding features to it a lot more often than I was in the last 3 months. Which means that sooner rather than later, it will become useful for someone besides me!

Anyway, we've got a new development box that's been in the works for a little while, with all the latest stuff. And since the current one is forcing me to wait longer for pages to load than the time I spend coding (literally), I'm taking a break from coding until we get it up - which will hopefully be soon.

Finally, if you're one of the few people who have downloaded and used cfrails, I'd like to hear from you if you've had performance issues. So let me know through the contact form until I get some comments up and running.