what we should learn from the emerald debacle

so the last week has seen a storm brewing in the second life community. at the heard of the storm is the popular emerald viewer from modular systems. drama erupted earlier this month when emerald developer LordGregGreg announced his departure from the project. it's not unusual for developers to leave open source projects, even "old skool" devs like LGG. this type of departure frequently goes unnoticed by the general public.

but what stoked the drama fires in this case was the reason LordGregGreg said he was leaving:

"I did not realize at the time that emkdu was added, that it could be used to add in code I was not able to see... Although replacing or deleting emkdu would resolve this issue, I also have to consider that this was hidden in the code for months without anyone knowing." --LGG

the "emkdu" code module referenced in this quote is a closed source component, and over the past several months there's been concern it's functions have been compromised. the issue is complicated and layered and has been used by some to refute the open source software development model.

but the core of the issue seems to be that the emerald project is too big for a single, trusted resource (like LordGregGreg) to effectively evaluate the trustworthiness of each check-in.

adding to emerald's woes was the allegation the third party viewer's HTML login page was maliciously mounting a Distributed Denial of Service (DDoS) attack on a rival software developer. modular system's response appeared by some to be "weak" and to not fully address the issue. wagner james au has a good write-up of the controversy at his new world notes blog.

potentially malicious code in open source projects? rogue devs DDoSing alleged bad guys? what's going on? i've read the cathedral and the bazaar several times and it seems to be implying FLOSS should be preventing these types of problems. after all, we're depending on open source methodologies to deliver on the libertarian promise of the technological meritocracy. the marketplace of ideas is supposed to encourage popular features and bug fixes while discounting the trivial and inconsequential.

free from the distortion of economic incentive, concepts are judged by their merits and implemented in software using the aggregated spare minutes of thousands of developers. but instead of being guided by adam smith's invisible hand, we seem to be avoiding the invisible foot affixed firmly in the metaphorical mouth.

lessons we can learn from the emerald debacle

1. with enough eyeballs, all bugs are certainly shallow. but this only works when you're careful about what you call a "bug." noted software security researcher john steven has a quote, "computers should do what you tell them to do, and only what you tell them to do." the implication here is that as an industry we expend a lot of effort on quality assurance; making sure that our software does what we think it should do. where we fall down as an industry is in software security; insuring that our software doesn't do what it's not supposed to do.

open source projects are not the only ones guilty of this. plenty of proprietary projects have inadvertently introduced vulnerabilities into their code. it's not easy to prove your software doesn't do something; it's proving a negative.

but the idea that the openness of a project will somehow reduce or eliminate security risks is magical thinking. so, lesson one is, "even if your software project is open, you still have to worry about software security."

2. order may spring from chaos, but there is no guarantee that it will. many software developers i work with have developed a misplaced faith in "emergent behaviors." in many instances, we see seemingly chaotic projects or processes "come together" while tightly controlled processes fail to accomplish their objectives. with respect to software development, or any large complicated human endeavor, i believe this stems from incomplete visibility.

no one participant has visibility into every event affecting the project. because we only see a fraction of the inputs and outputs, even "rational" processes seem random or chaotic. when these seemingly random events yield a successful outcome, it's easy to assume some "higher order" has emerged from the chaos.

and it probably has. but it's rarely the same higher order that you think emerged.

so lesson 2 is, "a development process that worked last time may not work this time."

3. perhaps the worst lesson from software projects, open or not, is that the efficacy of democracy to organize human activity is not universal. in other words, letting everyone's opinion carry equal weight in a technology project may be a bad idea. don't get me wrong, it's a great idea for local governments and i'm not saying that project leaders should act ruthlessly, ignoring the interests of project participants.

the problem with democracy on software projects is special interest may lead developers to introduce enhancements and bug fixes that are to the detriment of other developers or the general community.

for example, i run second life on a not-exactly high end system. i've got a reasonably beefy system, but i never run in "ultra" mode. it's just too much of a stress on my overly middle-of-the-road GPU. so why not just remove all that "graphics mode" clutter from preferences? it would certainly simplify a program with a reputation for being "not exactly easy to use."

i picked this ridiculous example for a reason. i don't think anyone would ever suggest removing features that degrade the experience of people who have gone to the trouble of buying high-end graphics hardware. but many projects have processes which could allow this to happen.

it's not exactly what happened in the emerald case, but it certainly seems emerald is lacking a single developer or architect who's role is to understand how all the pieces fit together and can proactively dissuade developers from doing things that would ultimately be harmful to other dev's efforts.

so lesson 3? "democracy is considered harmful in software projects."

4. the last, and perhaps the most important lesson might be the most depressing to software engineers. people frequently contribute to open source projects for the purpose of expressing a creative urge. many developers in the FLOSS community spend their days looking at proprietary code owned by a corporate entity. they come to open source projects to exercise their creative muscle in ways they find difficult in work.

open source projects, focused no the solution space rather than the problem space, allow a developer to solve problems without worrying about interference from marketing or sales. (okay, this isn't always the case, but i would argue it is most of the time.)

so this last lesson can be a bitter pill for some people: you can't escape process. sure, you can eliminate the sales department with their business motivations from the process; you can reduce the process to randomly shouting your intent to check in a module in a random IRC channel.

there are plenty of light-weight processes you can use. but you can't completely eliminate "process" and expect success. you must coordinate with other people. identifying a collection of best common practices and elevating them to the status of "process" will free you from having to think about how to communicate with your peers.

yes, process can constrain you. but the idea is it should constrain you in a way that is not offensive and in a way that will produce value.

lesson 4: appropriate process is a good thing; even for open source projects.

finally, maybe the most important lesson. everyone seems to flub this one at some point in their lives (myself included.) lesson 5 is "if you mess up, come clean early and fix your mistake(s)." the emerald team ignored this lesson when they tried to play down the DDoS attack on a rival's page. sure, maybe it really was a light hearted attempt at geek humor. but enough people didn't think it was funny.

and it's not just because it's the "right thing to do." even if you do make a completely innocent mistake, when a bunch of people jump down your throat, acknowledging the incident and apologizing for not seeing how it would affect other people will help disarm them.

these are just a few simple ideas; it's easy to be a "monday morning quarterback" for software projects. i am not trying to imply that the emerald dev team didn't try hard to maintain the security of their software. but it seems they may have been spending too little time on processes that may have been a little too weak.

there's no prescriptive advice in this post, so take it with a grain of salt. i won't pretend to tell you i know enough to tell you how to run your project without even knowing about it. but i like thinking about software, and these are a few things i've learned from hard experience.

your mileage may vary.