As the year grows to a close I find myself finishing up projects and preparing for the new year. I put together roughly 90,000 lines of code this year on a single project with one other developer, probably split somewhere around 80/20. It is, as I’ve told him, our boss and my wife, the best code I’ve ever designed and implemented. The things I’ve learned along the way would qualify for a series of masters, if I were so inclined to waste my time trying for such credibility.
Not all projects end before the holidays, of course. Most simply take a short hiatus and start back up in earnest in early January. What I do with the time between now and then is to get organized - again. I clean up build systems, update wikis and bug databases, do post-mortems on the work completed and use that to plan the work ahead.
As I started this process today it led me to think about what I’ve learned over my 20 years of software development and test experience. Much has changed in the industry and a lot has changed for me in the past 2 years. I would be remiss as an engineer if I didn’t share what I’ve learned with those who might be interested. With luck, maybe it will help them along as well.
So without further fanfare, here are the top 10 things I’ve learned about the software development industry over the past 20 years.
10. Java, the language, is good. JVM, the environment, is bad.
Whoever decided that adding a layer of software between an application and the operating system was the best solution for the sole purpose of preventing porting-based recompiles and complex #ifdefs must have been on crack. A JVM does nothing but, at best, limit your access to 30 years of operating system advances or, at worse, tries to reinvent the same. Java, on the other hand, is an extremely well designed object oriented language, far superior to the mess that is C++. As a C programmer for the majority of my career, I found it far easier to pick up Java (and even Perl’s object orientation) than to learn C++.
If we could just get native compilers for Java and remove the need for JNI (but leave the linkage to C libraries), Java would be the ideal development language for 90% of computing. With JVMs, however, it is useful to only 10% of computing.
9. The Web is overkill and a mess
Our industry took a major dive when the world discovered the Internet and decided it was the newest oil field that needed drilling. Of course, few of the money hungry bastards that paid for the drilling knew anything about this field. They didn’t know drilling into thin air gets you - well, air.
I will admit that you can do some amount of business on the Internet. I frequently make purchaes. But this is nothing more than direct buy purchasing. You eliminate brick-and-mortar costs, including distribution and middle-men. But the Internet in and of itself is not something to make money on. Even with internet telephony you use the backbone as a tranmission medium but still must provide services on top ot it. You still need a product. The Internet is not a product.
So the first mistake in this industry (or those we mistakenly let into our little club) was a lack of understanding of what the Internet is and what it can provide. The second was a misunderstanding of what the tools used to access this resource should be used for. Let me be perfectly clear here:
Not every application should be based in a browser.
Forget for a minute the vast security issues like cross-site scripting. A browser is exactly that - it’s used to view information. Interactivity was never part of the plan and the mechanisms available for interactivity are horrible hacks that have overly complicated whath should have been a moderately simple tool. More important, client side native applications can more effectively utilize 30 years of operating system experience and implementation (see previous rant on JVMs). Just because you can do something doesn’t mean it’s a good idea. You have to think about what you’re doing and why you want to do it.
Unfortunately, building browser-based applications is a fad that, like rap music, simply won’t die. That means remote systems that can support those applications. And that means - gulp and yuck, in one breath - Web Services.
My current work involves implementing a grid system over web services. The use of web services was decreed as part of the original contract for the product, which I was not involved with (though was hired just after the project started and was therefore stuck with it ad infinitum). Web Services has it’s good points. Synchronous communication allows for RPC-styled usage, for example. But over the network synchronous communication can be very slow.
We implement on top of JBoss because it’s free - plain and simple. The contract stipulated open source solutions. JBoss works quite well if you get your eggs all lined up. But configuration using XML is ridiculous.
XML as a data transfer protocol or as a data store format is not a bad idea. Utilizing existing parsers you can manage just about any incoming data and decide if you want to muck with it. But use the right tool for the job. XML is EVERYWHERE in Web Services. SOAP, for one, really gets in my way. Mixed with database transactions I have a hell of time sending the data the way I’ve been told i can (as serialized objects). Without diving into details I’ll say that Web Services is supposed to make interactivity with different applications easier but I find it does nothing but make things more complicated. What started out as a standardization effort has turned into an implementation nightmware. In the end, we got the server running the way we want it and avoid touching it at all costs. We do have an exposed Web Services interface but have wrapped that around a Java API. It’s much easier to use an API that try to decipher XML. God help us when we migrate to JBoss 5.
Don’t get me wrong, there are some good applications for the web out there. Wikis and blogs (specifically, Wordpress) are excellent targeted applications. But these are the exceptions, not the rule. And remember that wikies and blogs are not for trying to sell the Internet. They are for providing information. Exactly what the Internet was designed for - data dissemination.
8. Evolution, not revolution
Open source is being run by children. What they don’t understand, or wose don’t like, they reinvent. The GTK+ project recently ran into this problem. GNOME is being sucked into the Mono world for no valid reason. But my biggest gripe is probably with the change to init. Fedora (among others) recently started using upstart. The goal of that project is to speed boot up time. F10 uses this to parallelize boot processes. Personally, I see no difference. Speeding the boot process is a admirable goal but I think switching to upstart was more revolution than evolution and really didn’t address the problem.
But then maybe I should give them the benefit of the doubt and follow the three release rule. Every first release is a proof of concept. It shows that what you want to do can be done if not specifically the way its been done with the first release. The second release is a solid implementation based on sound planning that typically lacks important components that will, in the end, require major changes to the design. The third release is the one that will seriously evolve. Upstart, from the end user point of view, is still in the first release.
The three release rule has been very applicable to the many projects I’ve worked on over the years. We’re currently on release 2 of the grid software and I can point to areas that, while architected, were not well implemented and will require some redesign work next time around. Previous products included blade servers and network management software (reached release 3 but never had a detailed plan that could apply it directly to the three release rule), a robotic storage systm (never left release 1), an interactive cable system (never got past release 2 due to poor management vision), and operating systems (got stuck in release 2 before the project was killed by the morons at Dell). In every case the project itself evolved internally but only the operating system was truly evolutionary software. Everything else attempted to be revolutionary.
The problem with revolutionary is that people don’t like change. They want things to work they way they used to. Added complexity, even to improve performance, seldome goes over well with end users. You have to ease them into the transition. The less they see the transition the happier they are. Take the Linux desktop, for example. Why doesn’t it simply overtake the Windows desktop? It’s free. Isn’t that enough? No, it’s also different. Never mind if it works better or more efficiently. It simply works different. And most people are resistant to change.
You have to let software evolve. Revolution is a bloody, destructive process and can only succeed when evolution fails absolutely.
7. Patents suck
This one is obvious. If you write software you know you’ve never invented anything new. You’ve always taken another idea and expanded upon it. That is the true nature of software and it will never change. We all stand on the shoulders of giants. Patents make giants stand alone in their fields. Good for the giants. Bad for software. Patents suck.
The other big problem with patents is that they go completely against the grain of true engineers. Engineers revel in the knowledge the gain and they’re ability to share that knowledge. Patents hide knowledge. They limit engineers. Plain and simple. Patents are anti-knowledge.
6. Engineers can’t write. Or even communicate.
If you’re in school right now and wonder why they hell you have to take English classes I ask you to simply read all the manuals you have or can get your fingers on for any software you use. And try to match what you read with what you use. In most cases, there is no alignment. That’s because software engineers can’t write. They can’t communicate, period. They know very little about how to explain what they do, or have done, to others. They get flustered when others don’t just get it. In the end, the software developer creates something that no one knows what to do with.
Granted there are technical writers that have learned the fine art of translating geek-speak into human language. But that’s not enough. Software developers need to learn social skills. Starting with the spoken word, leading to the written word. Without this, you’ll never get past item #1 in this list.
5. If it’s too large to grasp, it’s too large period
If you can’t remember where in the code something happens, then you’ve built something that is too complex and will be too hard to maintain. This applies to C modules, Java classes, Makfiles and ant scripts, and any other type of code. Refactoring is the buzz word dujour for this problem, but good developers have a better way (see item #1).
If you write software, learn the unix way: small components linked together. Object oriented development helps here but the tools are only as good as the developer who uses them. Learn to break things down into the smallest pieces, and then build them up from there.
One of the things I’ve discovered recently in my coding is the use of plugins. Dynamic extensibility is a good way to add features, but it’s also a good way to patch a system. If everything is a plugin, then you need only update the plugin, not the entire system (except for those rare cases where the core plugin management needs to be updated). But the use of plugins requires you to think seriously about how data moves through the system and how various software components access that data. This requires, once again, item #1.
4. Stocks are bad. Bonuses are not salary.
C’mon guys. The dollar signs practically killed our industry in 2000. If you’re an engineer it should be obvious to you: stock holders know nothing about quality. If you work for a public company - leave immediately. Start your own business and never, and I mean NEVER, go public. Not if you want to create insanely great products with people who have the same vested interest in success that you do. Profits belong to the workers, not external shareholders. Direction should be determined by the company, not outside financial deal makers. If you’re in this business to get rich, you’re in it for the wrong reasons. We’re not doctors and we’re not lawyers. We actually love what we do and would do it for free (which explains the open source phenomenon to a great extent). If you wouldn’t do it for free, switch to Wall Street. They’re more you’re style.
And speaking of money, stop expecting bonuses. I worked for RLXTechnologies for a year and received bonuses every quarter. Very nice bonuses. From a company that had never, in seven years of angel investing and start up capitalization, made a dime. It’s no wonder they shuttered the place on December 23rd. Seriously. Two days before Christmas. If it seems to good to be true, it is. Bonuses are for when times are good. Save the money for the times when they’re not.
More important, make sure you work for someone who understands this. I work for someone who isn’t taking a salary yet. He’s building the company. He has a plan and it doesn’t include investors. He has a plan for the people who work for him. This is a good thing. A very good thing. It is the sole thing makes working on government projects tolerable. You want to know why the auto industry is failing? Because the people on the front lines didn’t pay attention to what the people in charge understood. The people on the front lines are just as much to blame for the industry’s failures. In their unions they had the power to make changes. But they didn’t. They simply protected their salaries and jobs. That’s not enough. You have to take a more active role. And you have to work for people who know that.
3. IDEs are crutches
I won’t belabor this point. Just get your hands dirty. Drag and drop coding teaches you nothing. If you don’t understand why things work, you’re only adding paint to a system. You’re aren’t reinforcing its infrastructure.
2. The code is never complete. Ship it anyway.
If you haven’t tried coding then you won’t understand this. But the idea that something that didn’t exist before and can control the outside world (physically, financially or emotionally) can be created by you simply by typing is mind blowing. Tweaking is never ending. The code is never complete. You’ll always find something more to do to it. That’s the great part.
The bad part is that your code is never complete. Unless you made serious plans, you don’t know when you can ship it. The users never know what they’re getting. And you, great developer that you are, are sucked into the rot of feature creep.
The only way around this is a plan. Developers need plans. Detailed, mind numbing, wordy and over diagramed plans.
1. Plan first, code later
Real project development is 90% planning, 10% coding. All those ideas about extreme programming are worthless without a plan. You can still code, try, recode, try again - in fact during implementation there is nothing to prevent using extreme programming ideas (if that’s what gets you off - it doesn’t me). But coding is the smallest part of the problem. If you have no idea where you’re headed or have any idea how long it’ll take to get there then you’re running on treadmill. You get lots of work done but you get nowhere.
My first job was at BNR, now Nortel, as a software test engineer for a data communications product (that eventually died without seeing the light of day). I was taught to write a high level test plan describing what I hoped to achieve followed by detailed level test plans that implemented the actual tests. That first job taught me how important planning really is. Since then, I’ve always sketched out what I was trying to accomplish before writing any code. This wasn’t always detailed or formalized, but recently it has gotten extremly formal thanks, in part, to the use of wikis for project planning. I create diagrams (with Dia) to define software components, the outline a high level design to explain these componnents and their interaction. Finally, a detailed architecture explains how each component should be implemented, including external resources that should be intergrated with them.
This kind of planning makes coding very easy. And while I say that the planning is 90%, I mean that only in the amount of serious though that is required. You don’t necessarily code faster, you just code better because you know what you’re trying to achieve. In mind numbing detail, you know what you’re trying to achieve.
Though I’m pushing the importance of the plan, I need to make it clear that the plan has to be flexible. If you can’t change the architecture you might code yourself into a corner and end up either breaking the three release rule or never reaching release 3. So make a plan, and stick to it. But only as much as is reasonably necessary.
Related posts
