A DB2 Person in an Agile Software Development Discussion

I'm posting this week from Orlando, Florida, USA, site of the winter meeting of the SHARE user group (www.share.org). I came here primarily for the purpose of delivering a presentation at the conference (the topic was SOA for DB2 people), but when I attend a conference as a speaker I also try to sit in on some presentations delivered by others (never stop learning, I say). Thus it was that I made my way yesterday afternoon to a session on agile software development. As it turned out, the speaker couldn't attend the conference due to a last-minute travel-related problem. I was thinking that we'd just walk, but the session moderator asked us if we wanted to spend the hour in an informal "free-for-all" discussion centered on agile development. Most of the people in the room liked that idea, and the ensuing discussion actually turned out to be, for me, one the highlights of my day at the conference.


Most of the people in the room were, as you might imagine, software developers. I was the only database person there, and I fessed up to that early on. We had some interesting conversations about the ways in which database people can help their coding colleagues who are wanting to do their thing in an agile way. I'm not an expert on the subject, but I'd say that agile development is largely about iterative development of an application, with lots of smaller-scale releases versus one or very few great big releases. It's also characterized by a cross-functional team-based approach to development, with differently-skilled people (not just programmers, but folks with specialized skills in testing, data, requirements analysis, etc., as well) working together to get each iterative release ready within the defined (usually pretty short) time frame (a good overview of agile software development can be found in wikipedia, at www.wikipedia.org). With respect to the role of data people in an agile development environment, this is what I heard from programmers during our impromptu "birds-of-a-feather" session:

• Database people are very much welcome at the table. All too often, an organization's database specialists are not involved up-front in the development of new applications, and this is not just a matter of application programmers and architects not issuing invitations. Frequently, it's DBAs themselves who pass on opportunities ("not my job"), or it's an IT organizational structure that has DBAs working only to put together development databases in response to requirements that are "thrown over the transom" by programming groups. Sometimes a DBA has to take it upon himself (or herself) to get engaged early on with developers working on an agile application development project, but if that initiative is taken, chances are the programmers and architects will be very pleasantly surprised.
  
• Speed is of the essence. The programmers might need a data store, but what they want is something done in a hurry. The database developed to facilitate agile development of an application doesn't have to be designed to the nth degree with the production environment in mind. Get something out there fast, and make schema changes (or what have you) as you deem advisable as you go along. Asking the developers to hold off on writing code until a really rigorous logical and physical database design is completed is probably not going to cut it. That's the heavy, old way of doing things.
• Schema abstraction is very much appreciated. The developers want a database, but they'd rather not have to know about the underlying schema. There are various ways to abstract the application-database interface. One way is to package the SQL in server-side stored procedures. These can be called by all kinds of programs in all kinds of environments (definitely including Java and .NET), and to the calling programs DB2 stored procedures don't look any different from stored procedures in other DBMS environments. An even better step might be to expose the stored procedures as Web services (note that this can be done quite readily using IBM's new Data Studio product, which can be downloaded for free if you're going to use it for development purposes - see http://www-306.ibm.com/software/data/studio/). In any case, abstraction is your friend, too, because it enables you to make those database design changes you want to make (perhaps to improve application performance) in a behind-the-scenes way, without changing the interface to the data as the programmers see it.

Some of the other interesting comments made by session participants had to do with the CULTURAL change needed within many organizations to support an agile programming environment. You see, the traditional application development process has people working in a highly specialized way as though they were standing by an assembly line in a manufacturing plant. You know, coders code, database people design databases, testers test, and so on, but instead of working in cross-functional teams they man their posts and work on the stuff that comes their way. It's like put on the tires, put on the tires, put one the tires, etc. in a car factory. The work that goes on ahead of what you do (up the assembly line) and after what you do (down the line) is not your concern. You just make sure that the tires go on right. That approach may create comfort zones for some people ("put the tires on right and you'll be OK"), but it can lead to tunnel vision and to communication that flows too much in a vertical way ("I just do what my manager tells me to do") and too little in a horizontal way. People feel less personally invested in the finished product, and that can result in people not performing up to their potential since breakthrough results often rely in large part on the degree to which people feel an emotional attachment to what they do (i.e., when work really means something to people, work quality and productivity can soar).

The cultural change that may have to precede the establishment of a successful agile software development environment may be analogous to the revolutionary cross-functional team approach to automobile manufacturing introduced by Japanese car companies two or three decades ago. The face-to-face work done by cohesive groups of differently-skilled people engaged in the start-to-finish building of automobiles had a lot to do with the quantum leaps in quality, efficiency, and innovation seen in those factories. It set the standard that the rest of the auto-manufacturing world subsequently had to work towards. Similarly, agile development can be hindered, if not outright emasculated, by an emphasis within IT organizations on hierarchy, turf definition and protection, and work silo-ization. Of course, big-time organizational change has to be led from the top, but that can be challenging because many at the top are products of the legacy way of doing things. In large part, change may come down to trust: can technical professionals be trusted to perform as needed in roles that are less rigid and less precisely defined than the historical norm, with more floor-level leadership and initiative and less in the way of management-imposed structure? Personally, I believe that the rewards to an organization that can come from extending that kind of trust to the IT tech folks - from giving cross-functional teams an end goal and setting them lose to achieve it using their own creativity and drive - can be enormous.

Not a bad way to spend an hour at a conference, eh?