“Pride cometh before a fall.” -- I use the term “markitecture” to encapsulate the tendency of vendors to oversimplify or even overly aggrandize the capability of a software system and leave clients without all the facts. There is another side of the markitecture coin and that is the tendency of software professionals to accept a systems markitecture without validating the actual architecture and related metrics and facts that help make better informed decisions.
Dr. Liu’s System
I will say that I personally fell prey to markitecture early on in my career which I think made me a better evaluator of architectures (thankfully). As I mentioned in the “t-off” war stories post, I’ve been using Oracle since version 4 in 1985 [actually, Tom Siebel was our Oracle salesman who was unhappy when Larry Ellison told my colleague, Demetrious K., “We’ll match their price!” as Demetrious was explaining to Larry that we might go with Ingress since it was cheaper; but that’s another story]. Actually, Oracle was very impressive in 1985, in part, because it ran on so many platforms. We in the R&D facility were running VAX/VMS and PC’s and corporate were running mainframes, so Oracle was an easier sell to corporate.
So anyway, we come to Dr. Liu’s system (my second Oracle application). I was tasked to replace his micro computer application (for counting dead insects that were exposed to various compounds) that was written in C under the CP/M operating system, with an Oracle application running on a PC under MS/DOS. The target platform was a DEC Rainbow 8086 machine with 8-megahertz clock speed, a 5-meg hard drive, and recently upgraded to 640K of memory [yes, Oracle ran under that configuration]. Well, at them time, I didn’t really like PC’s and thought the world of the VAXes, so in my infinite wisdom (I graduated with a BS in Computer Science in 1984) I thought to myself, “well, since Oracle runs on VAXes and PC’s, I’ll do my development in the platform I like and just port it to the PC later”. My development platform was single user on a VAX 8600 (as I recall); the system was working and reports took just a second to run, so I figured all would be well when I ported the app to the PC. Oh, I forgot to say, the system was fully normalized such that C.J.Date and E.F.Codd would be proud.
The Day of the Demo
Since this was the first Oracle application for our company going on a PC, my management was keen to see a demo. I ported the application to the PC the day of the demo but had not had a chance to fully test everything before the demo. So here I was in my office with all the management behind me watching as I demo’d Dr. Liu’s system. All was going well until it came the time to run the report – I hit return to run the report, and the seconds I experienced on the VAX was not materializing (and it was getting hot in the room [or was it just me?]). I loosened my tie for some breathing room [yes, we wore a suit and tie back in the day] and after several minutes of the classical song and dance routine one does in these situations, the head honcho pats me on the back and says, “Looks like you have some more work to do” and leaves the room. One-by-one, from the highest to the lowest manager, they all gradually leak out of the room, parting with some words of encouragement … all except Demetrious who pulled up along side of me and said, “let’s roll up our sleeves and get to work!”. First of all, I can’t say enough about how valuable it was to me in my career to have such a great mentor who really knew his stuff. By the way, the report eventually ran, but it took 45 minutes on the PC.
Denormalizing and Replatforming
Of course, back in V4 of Oracle there were no v$ tables or bstat and estat, or StatsPack, or AWR, so one was left with very few diagnostic tools [actually none]. Demetrious’s first suggestion was to denormalize the application so it would do less joins for the report; we denormalized like crazy. This helped quite a bit as we were able to get the 45-minute report down to 12 minutes. We then replatformed to an IBM 80286 machine with a whopping 12-megahertz clock speed and was able to get the report down to 8 minutes.
The Final Rejection by Dr. Liu
Now that the system is running the best we could get it to run and I’m all proud of my accomplishment [remember: “Pride cometh before a fall”], I took the PC to Dr. Liu’s lab and demo it to him. Well, Dr. Liu said some nice things about the system but told me he wasn’t going to use it [I thought he “had” to use it]. So, Dr. Liu shows me the original system (written in C on CP/M OS) and the same report took 3 minutes on that system. Dr. Liu was not going to accept a system that was no better that the system that it was intended to replace.
Some Lesions
[Edit: I meant to say LESSONS, but lesions/cuts fits here too]
- The system is not acceptable if it does not do what the user reasonably expects it to do. The only debate is what is “reasonable”.
- Although my second Oracle application was technically a failure, there were lots of lessons which carried forward, such as don’t just blindly accept the markitecture; you must evaluate all the relevant facts and anticipate the impact of the actual architecture on your system.
- Capacity planning and benchmarking is an essential notion that should be actioned in your project. There is another war story regarding the state income tax system that was developed for micro-vaxes without any benchmarking or capacity planning; the long and short of is the system couldn’t handle the workload at the scale of an actual income tax filing season and had to be reverted to the old system [the state canceled the project and let go all their Oracle developers].
- Denormalizing does work, but it makes your system more brittle and less adaptable to when the requirements change.
- Having a good mentor like Demetrious is a life changing experience.
- Start building your bag of tricks early.
Appeal for More War Stories
That’s it for now. I’m starting to get volunteers to join me in sharing their war stories, so tune up your war stories (I’m toying with the notion of a podcast format). My vision here is to make a place where DBA’s, developers, Oracle “grey beards” and other battle-scarred veteran (and new) techies get together and tell each other their war stories and have some fun. Please contact me if you are interested in having your war story included. Of course, “the stories you are about to hear are true, only the names have been changed to protect the innocent” [some of you old timers will recall where this quote is from].