Language, semantics and software development

I just read Robert Barta's hilarious post on some of Java's (the programming language) antics, which, for those who's following from home, I agree with wholeheartedly. His post triggered a monologue in my head as I was going out for coffee today, and I felt the need to jot down what I think is my thoughts on languages, semantics and semiotics from that inner tirade.

No, this won't be on programming languages, at least not only. Language permutes every aspect of our human evolvement, and has been one of the main factors for our success for the last 100 000 years or more, and lays down the foundations for that one thing which everything else spins around; communication.

We communicate. We all do it, all the time, every day. In fact, apart from physically mundane things such as eating, resting and procreating, it is all we do.

But as much as we do it all the time, and as such you would expect us to get better at it, we need more than just communication, but the right kind of communication. Anything can communicate, but anything can't always communicate well. A dog barks at me, but I'm not always sure what that bark means (although the drool and glistening fangs provide valuable hints). Good communication is how to get the message across without ambiguity.

Developing software

Let's take a rather common problem; how do we develop software? Well, it's actually quite easy. Just open whatever tool you need, start writing code, hit "compile", and voila! The trick is of course not to develop software, but to develop software well, to remove the ambiguity between the tools we use and the customer who asked for it. And just like with communication, we expect that if we do this often and repeatedly, we learn something along the way, and get better at it.

But developing software still sucks. I've been in this business for almost 20 years working in more than 7 different countries, and every place had the exact same problem; how do we get the communication right?

There's an absolutely stunning focus amongst people in my line of work on the technological parts of the solutions we create. Of course we need those technological bits to make stuff happen, but most of the time they drive the solution more than the other way around. There's a sensible saying that says "choose the right tool for the job", but it seems to me that very few actually stop and ponder that very good piece of advice. What does it mean to choose the "right tool for the job"?

The job

There's a few indicators to the problem above. First of all, what is "the job"?

Our ideal target is a working functionally-complete system that solves some loosely defined problem. As professionals we put a lot of methodology and effort into translating that napkin sketch into documentation fit for enterprise development, which in turn will be translated into programming code, which in turn a GUI will translate to the users of the system. We can say with great confidence that "the job" is pretty much a translation job.

How many levels of translation can we find in our daily work? Let's see, first there's Bob who has an idea (level 1) who talks to an agreeing Pete (level 2), they talk to John (level 3) who agrees (level 4) who writes a mail (level 5) to the marketing manager (level 6) who brings that to a meeting (level 7) to get agreement (level 8). Then John takes the revised document (level 9) to the CEO (level 10) who agrees (level 11) and gives a go, to which the document is sent to Sam who will write a spec (level 12) which Bob, Pete, John, CEO, Marketing boss and Sam all agree on matches the original idea (level 13). Then the spec is sent to the development project manager Jim (level 14) for scheduling, and then to the development team (level 15) for estimation and thoughts (level 16). they come up with estimations and suggestions (level 17), there's a bit back and forth between them and marketing (level 18 through 22), before Jill the functional designer is given the roughly agreed upon documents (level 23) which she alters (level 24) and the leadership team agrees with (level 25). The document now goes back to the development team approved (level 26), and entered their schedule. Development starts with visual design sketches by Pop (level 27), which needs approval (level 28 through 34), which Jim further plans (level 35), and then each developer gets a slice of the new plan (level 36 through 40). A couple of iterations go by (level 41 through 46), and we're ready for development release 1 (level 47) which needs to be agreed upon by all bosses involved (level 48), with tweaks and continued development iterations ongoing (level 49 through 52), we're getting to first test version (level 53), which needs agreement from bosses (level 54 through 57), and finally the first production release that is to match Bob's idea (level 58), to which Bob says "WTF?" (level 59) and to which customers respond "WTF?" (level 60 through 65)

You get the picture. And it gets worse; not only must we translate from person to person, but from person to tool to person to tool to tool to tool to person to tool to person to person to tool to tool ... and on and on and on.

The tools

It's hence a given that our tools need to be good at translations between humans and computers, no? The whole idea of programming languages is to do just this, so choose your language wisely, young padawan. Some choose low-level languages to get closer to how the computer sees things, other choose high-level languages hoping those who built the underlying layers have all the semantics down pat. Or up.

Back to Robert's post; in Java people use a lot of syntax to express something that in other languages takes less. But is language about semantics? Is the difference between French and German the words they use? Of course not. The same goes for programming languages.

How many levels of translation do I find in my development stack these days? I'm doing a pretty straight-forward webapp wizard thing. We could start with the customer having an old webapp which sucks, and they want a new and improved one. So they tell us what they want (level 1) which we jot down on paper (level 2) while showing us the old one (level 3), of which we agree roughly on what needs to be done (level 4). Our functional designer draws up a few sketches (level 5), which the customer representative agrees to (level 6) which she takes up the ladder to get more agreement on (level 7), and we're handed responses (level 8) to which we update the sketches (level 9) ... and on and on and on.

But let's look at it strictly from a technical aspect. Let's pretend we have a joint human understanding of what the needs and solutions should be. First, there's legacy to look at, and here we've got Java all the way (level 1), running WebSphere app servers (level 2), using enterprise given code guides shaped with Java 1.6 / 6 (level 3), running in three environments (level 4), with Maven to build (level 5), Hibernate for db (level 6), Struts 2.0 for MVC (level 7) and some JSP semantics (level 8), FileTags for more JSP "fun" (level 9), and of course JSP itself with mixed in Java code (level 10), running Jetty in development (level 11), with proxied Maven repositories due to no internet connection in devel (!!) (level 12), our own JavaBean naming conventions (level 13), legacy-based property names (level 14), about 10 different Java tools that comes bundled (but not used) with the platform (level 15, just in case namespaces gets muddled) ... and on and on and on. And that's just development. I could add a cut-down version of Eclipse as IDE (levels 16 through 20), running through WM Ware (level 21) and Citrix for intranet access (level 22), servers running Linux (level 23 through 25) while we develop on Windows XP (level 26 through 28) ...

APIAPIAPIAPI?

You know, at this point we gotta mentions application programming interface (API), because, frankly, they drive me friggin' nuts! Basically they are lists of words (which are objects or functions) that convey semantics of whatever module they represent. Sorry, that was the interpretive version. I mean, they are "a set of declarations of the functions (or procedures) that an operating system, library or service provides to support requests made by computer programs." It's what we use - all the time, all over the place - to have bits of software talk to other bits. All the semantics (i.e. meaning) is left in the hands of those write the docs for those APIs, which must match the behaviour of the code. Without this perfect match of perfect symmetry, everything can quite quickly turn to batshit. And it does, because this is what programmers really do all day; try to use APIs the way they were intended. Communication, programmer to programmer. But the focus again is on technology, rarely on the meaning of the thing.

Did I mention that I hate automated documentation with a passion? The way an API works is left in the hands of programmers slapping keywords with short explanations in code? All the hundreds of layers of fragile communication is autogenerated from code?

I refer often to something I call API thinking, where the semantics of a problem space is defined through keywords. It's a bit like trying to explain the complex nature of the Iraq War (the current one) in terms of War, Bush, 1991, 2001, USA, Iraq, Oil, WMD, and few getters and setters on top. Let's call it the iWreck API ;

Dependency oil = new Dependency ( Dependency::RESOURCE_OIL ) ;
int startDate = new Date ( "2001" ) ;
War war = new War ( startDate, War::TIME_UNKNOWN ) ;
war.author = "Bush" ;
war.history = new HistoryFactory ( new HistoryContext ( new Context ( new Identifier ( ... ) ) ) ) ) ) ;

There, done. Easy-peasy. Now we know the meaning of the Iraq War (current). What else is there to say? Come to think of it, I reckon Bush would make an excellent programmer.

Stacking it up

How many levels of translating our communication from one unit to the other have we got when we add them all up, all the human ones, the tools and the systems? Bucketloads! So many it is quite crazy to think that any of the original ideas come through this enormous filter monster we have created!

Doing communication right is damn hard. How can I explain? Maybe through example. I had some JSP code similar to this ;

<framework>
<section>
<sub-section />
<sub-section />
<sub-section />
</section>
</framework>

When someone was going through the code for this, they found out that the JSP for the

tag didn't add any HTML nor did any logic in it, but simply told the compiler to travel up the stack. They hence removed

tags, because they didn't have any functionality attached. We now got;

<framework>
<sub-section />
<sub-section />
<sub-section />
</framework>

I'm not sure you've noticed, but you just lost the semantics between the <framework> and a <sub-section>. What is a sub-section without a section? It's API hell, that's what. It may not look like much of a deal to most people, but remember here what we're trying to do, remember that very thing, the most important thing in all that we do; communication! And when you're removing a section in which a sub-section lives, you get a sub-section out of context. The reason for the sub-sections' life is lost, and the technologists killed it thinking they cleared things up by removing redundant code.

My soul weeps. I see this sort of innocent but utterly devestating loss of semantics through every level of interpretation, from customer to developer to user. We remove little bits here and there, add a bit there, and the original idea comes out all warped and garbled, and we think it is a good thing!

Folks, if I have one request if you were to listen to anything, then please be careful about semantics and meaning, careful about adding it in, state purpose and meaning, write more about what the hell the point of what you're trying to achive, in your APIs, in code, in thinking, in documenting, in specing, in talking to eachother, in anything you do, and - perhaps and hopefully poignantly - in reading my little plea for semantic sanity in software development.

Thoughts on PHP

Yes, yes, I admit the herasy; I like PHP. No, no, PHP has tons of worts with it, so no, it's not better that alternative X, Y or Z for task Q, W or E. I hate comparing languages this way, feature by feature, syntax by semantics, and so on. I like to judge languages on two things;

Environment

I really like the CGI style for web resources. No, PHP ain't CGI (unless you're in a pain-self-inflicting mood) but most often a glorious Apache module which reuse all the goodness it can offer, but the model of a totally independent scripting engine which needs to mold its relationships, and then you throw it away when you're done, makes for a clean, fast and very scalable framework. PHP basically compiles together mostly C modules, and use its simple syntax to glue stuff together. Yes, there's a backlog of really shitty and badly written code out there, especially when people have no clue. And when the threshold is as low as it is for PHP, that's inevitable, but I hardly hold that against the language itself. The environment in which that shitty code runs is really good. To me, the environment is the best part of PHP. Perl falls somewhat into this same category.

For those in the know, this model is very similar to how a RESTful system works, where the interpreter is a manifestation of a resource. In resource-oriented development this means gold, and is very important to me as my environment supports my RESTful way of designing systems.

Style

And since style is something any good developer can control themselves (unless their language is super-strict), this really is the main thing that I like about PHP; It gives me the freedom to make things work for me. I can choose the methodology, whether a function, class or inline is best suited, and since PHP always is evaluated in run-time, I can make my environment depend on real-time parameters rather than a pre-compiled utopia.

Zend Framework

I've been using the Zend Framework for the last year or so, and it's a great framework as such, although the focus has been mostly to put OO wrappers around common PHP idioms and conventions. As such it works great to perhaps consolidate the features, and perhaps give PHP 6 a future direction. This is the best part of the Zend Framework.

The bad thing about the Zend Framework is that it imposes its own style, and somewhat alters the environment. Ouch, on both the things that make PHP my choice tool. I've struggled quite a bit in trying to reuse certain parts of the framework, extend others, and generally use some bits without using the whole shebang. There's not a lot of dependencies between the components, but just enough to make it tricky to do serious stuff (for example writing an alternative "threads-like" HTTP adapter to HTTP Request).

Now, people talk about the Zend Frameworks impact on the future of the PHP language. Yes, one can always hope that this is the case, but I think people are a little too preoccupied with the OO capabilities and forgetting perhaps what makes PHP really popular with those who choose to continue to use it past their first few applications. Do not fall into the trap thinking OO is somehow better than other ways.

As much as people think MVC is the best thing, I really don't care about that. MVC works great for some things, not for all (for example, I have a REST framework that use a completely different model, a more resource oriented model), so to impose MVC as the modus operandi is not good, and indeed something that makes reuse of other ZF modules a bit trickier. Instead of a MVC focus, there should be a strong highlighting of a uniform interface to the environment itself. It's the environment that's cool with PHP, so let's make interfacing to that better. There's some work on wrappers and readers for various aspects of the HTTP protocol, but only the most basic stuff is in there and needs serious work. With PHP I could do serious HTTP applications; with Zend Framework I'm limited, and need to hack and extend. Let's get HTTP savvy, not MVC drones.

Availability and diversity

PHP is everywhere. Period. I can use pretty much any ISP or in-house hosting to host most of what I need. And there are so many different open-source projects about that uses PHP (WikiPedia, WordPress, PHPBB, PHPMyAdmin, Drupal, Joomla, Flickr ... the list goes on) meaning both hosting and the amount of high-quality tools are abundant. The LAMP stack is pretty much supported on every hardware and software platform. Heck, I can even run it on the JVM.

I have written many tools over the years, some good, some bad, but I'm always happy to find out that I can copy my old files into any new environment and they will pretty much always run straight away, no tweaking. I can't tell you how important this is!

The shitty stuff

No system is perfect, to some specific definition of what "perfect" means. With PHP, I've learned to live with its odd bits, such as funny booleans and comparators, diverse and non-uniform APIs, sloppy exception handling, no shared memory (which may or may not be a good thing, but certainly a big part of its design), and a few syntactic snuffs (Why not introduce "$$" as a shortcut for "$this->", for example? Tidbits).

Some say that PHP code is crap, usually said when encountering - indeed! - shitty code. But shitty code is everywhere. Even in the most tied-down, static and controlled environments you still get shit. Some say less shit. I say shit with a different color, not different odor. I've been programming for almost 30 years soon, and let me tell you, I've encountered shitty code in every single environment and language I've ever tried, and I'm one of those who tries every language out there. There is no language who can hold a stick to the wonderful imaginitive mind of man, able to lay bricks where diamonds should have shone.

Some say the PHP language itself is immature, or unstructured, or some other parameter that their favourite language holds. No, sorry, but there's not that much which separates all our different languages (except BrainFuck ... that one is seriously different. :), and indeed a lot of the language wars are really more about their API designs than the languages themselves. For example, the syntax of Java is tolerable, but a lot of its APIs are not. The syntax of Python is ok, but some of its APIs are great. Ruby has good syntax, but to me some confusing APIs.

It's really mostly about APIs, and how we create models to solve our problems. It's all about models, as APIs in themselves are models. The API is not the language.

Round off

I should here of course mention the link between models and Topic Maps, as the latter is a way to define, work with and exchange / share the former. Couple this model openness with resource-orientation, and I think it makes for a very interesting environement. But as this whole shebang is part of the new framework I'm working on, expect more blogging on this in the very near future. Have a wonderful week.

Parting thoughts

It's Friday, and I'm leaving my computer alone over the weekend. It's here at the end of the week I thought I'd give a quick status update on things big and small ;

• Sam's first steps;
  
  
• Some more family tidbits; Lilje just turned 5, and started barnehage (kindergarten) which she just loves! Grace is going great at school, with lots of friends, and she now speaks a very tolerable Norwegian. We've started to use Norwegian as our main means of talking these days. Sam is a social little rascal, and now of course wants to walk everywhere. Julie is still too far away from her family to really be happy, but at least we have a few favourite things to make our exsistance herer good, such as solskinnsbolle (a kind of layered horizontal whirly bun, with a yummy egg-cream center) at Godt Brød (great littel bakery), coffee at Kaffepikene (the absolutely, without a doubt, the best darn coffee in town!), norwegian bread (how can we ever go back?) and ice-cream. Oh, and my beloved woods. I can't rave enough about the woods! We all love it. We all love Norwegian nature; it's simply stunning, even the simple stuff.
  
• My RESTful Topic Maps framework lumbers ahead. I'll be using it for two major real-world projects, and will verify its success or failure from that. I'm sure it will be tweaked a lot.
• The kids are doing well, the parents not so much; the other side of the planet is somewhat problematic, as our values and views have changed so much in the last 5 years. The Norway I came back to isn't the Norway I left. We're contemplating leaving for Australia sooner than first thought.
• My eyes have been a painful nightmare the last 6 or 7 years, slowly but surely disintegrating my quality of life (and, sadly, those around me; my nickname lately is "grumpy". I'm so sorry!) where either my glasses have caused a pressure and hence headaches, or the lack of glasses makes me squint to the point of getting headaches, or contacts itch my eyes so much that I fall asleep (probably with a headache). Last month it became unbearable, with headaches (as in migraines; really painful headaches with a center right inside my eyes), and I didn't know what to do. I've so far seen 7 eye specialists (4 in Australia, 3 in Norway), including a full scan of my head and sinuses. Nothing. Until last week, where in a casual conversation between my sister and my wife, the option of allergies to pollen came up, and lo and behold, there seemed to be a pattern match between my worst periods of headaches with pollen outbreaks. Last Sunday I raced to the pharmacy and got anti-histamine pills, and - ladies and gentlemen - I haven't had a single headache since; the pills probably reduce the inflammation in certain pressure-points (top of nose, and the forehead) so the headaches don't develop. As you can imagine, I am over the moon about this result, and I hope there might be some answers as I get myself to a doctor in July for a full checkup on this.
• I'm doing an interesting UCD project on the side. I had forgotten how much fun and enjoyment I get from facilitating projects and meetings. I have of late immersed myself in coaching (course given through work; I've been doing stuff like this for years, but this time I've formalized it a bit more, and getting a Norwegian translation of the stuff) which seems to come naturally to me. Presentations, meetings, sessions ... I'm having a blast, with good results to boot.

Ok, have a great weekend. See you on the other side.