[I started writing this in 2016 and unearthed it amongst some old drafts. But 6 years have only intensified my feelings here, so here it is updated and finished]
I’m sure you’ve all heard the statement from Arthur C. Clark that “Any sufficiently advanced technology is indistinguishable from magic.” But I had an exchange which convinced me of a variation of that: “Any suffciently hyped technology is indistinguishable from religion.”
The case in point was a discussion with someone who seemed to think that Git was the only version control system which had checkin hooks. And after informing him that every modern version control system, indeed, every one of them I’ve used in the last two decades, has such things (in one way or another), he repeated the same thing later on in the conversation, as if unable to process this new piece of information which contradicted established dogma.
Another interaction was with someone who asserted that Git was “more secure”. When I questioned him as to exactly how it is more secure, we was unable to articulate anything meaningful. Then I pointed out that is was trivial to forge checkins (and even demonstrated it in front of him by doing a checkin in his name), but this didn’t phase him, and he returned, mindlessly, to his original point.
I have nothing against Git on the whole, I use it myself every day. I have minor gripes with it, mainly having to do with the arcane, counter-intuitive interface (like this). But my biggest gripe is the religious fervor with which it is hyped, the irrational one-size-fits-all, Maslow’s Hammer weilding, be-all and end-all, the perfect final pinacle of version control. For ever and ever. Amen!
There have been many “religious wars” in the software world over the years. Long ago I defected to the Emacs camp, so I know it well. But with all of those wars, there were always competing technologies; differing views on how to approach a problem. But in this case, there is only one left, all others have been shouted down into irrelevancy. At the time of Git’s ascendance, I was managing at least 6 different version control systems, and I thought this would just be one more to add to the mix. But I was mystified as my team was quickly sidelined as everyone mindlessly rushed to Git.
I am a believer in using the right tool for the right job, and Git is certainly the right tool in a lot of cases, but not every problem is a nail. Sometimes you need different tools.
I notice that I wrote about this earlier, but I am now taking the long view of this: every ascendant technology eventually declines as the next shiny thing attracts everyone’s attention. I look forward to the day something new comes along and pushes Git to the sidelines.
I ran into a situation today, which was quite astonishing: Git creates new repositories in an inconsistent state. Until the first checkin is done, the repository has HEAD pointing to a nonexistent location. I discovered this because I was replicating several other team’s Git servers for backup purposes. In experimenting with this I came up with this reproduction:
$ git init --bare foo.git
Initialized empty Git repository in /tmp/foo.git/
$ git clone --bare foo.git foo2.git
Cloning into bare repository 'foo2.git'...
warning: You appear to have cloned an empty repository.
$ cd foo2.git
$ git fetch
fatal: Couldn't find remote ref HEAD
fatal: The remote end hung up unexpectedly
While doing clone and fetch from an empty repository is a silly thing to do, but it isn’t worthy of a fatal error. No other systems I work with have this flaw. So now I have to modify my replication scripts to detect such repositories and avoid them.
Welcome to my reworked blog. This will mark the third incarnation of this blog. There was a short-lived Drupal based site, then I set up Wordpress 6 years ago.
So what’s that you say? It sounded like you said “so what”, but I know you really meant “how come?”
About a month ago I get an email from the good people at Laughing Squid informed me that I had vastly outstripped my compute cycle quota. I thought this was rather surprising given that few people ever read this blog, let alone comment. There had been a number of attempts by spammers to post comments on the site, but, at worst there were only a dozen or so a day.
My first thought was that someone had broken in and uploaded their own PHP to use my compute cycles for their nefarious purposes (this happened several years ago). But several searches for such things turned up nothing. Unfortunately, there is no way to determine what is consuming the compute cycles, or at least I presume so, as my repeated questions about this went unanswered. Furthermore the apache logs are not provided in real-time nor is the compute cycle accounting, so it was going to take a lot of guesswork to fix this. Poking through the apache logs revealed that thousands of hits every day (99% of my traffic) was to wp-login.php. The only logical conclusion was that somebody was trying to break in, and, in the process, caused wp-login.php to consume a lot of compute cycles encrypting bogus passwords. I tried a plugin to block repeated attempts, but didn’t help all that much (it reduced compute cycles by 50% where my goal was 99%). So, I tried the brute-force method, I renamed the login script so that all such attempts would be immediately refused. Viola! Everything went back to normal.
Now all of this started to make me think: why am I bothering with Wordpress? What does it give me? It lets me edit upcoming posts from anywhere on the internet (though I have rarely taken advantage of that). It lets people comment on my posts (the number of times that has happened can be counted on one hand). But the downside is that I now have to monitor the version of Wordpress and keep updating it to keep up with security fixes (failing to do that several years back earned me a break-in). I have to monitor the comments queue and reject spam. Wordpress uses up a lot of disk space, MySQL is a hassle to maintain, and people with nefarious intent can easily create havok by running me over my quotas. And on top of that there are no tools for diagnosing when this happens.
So, it wasn’t worth it. Static html doesn’t use compute cycles or can it be hacked. I started looking into static blog generators. I had considered Bloxsom many years earlier before settling on Wordpress. Sadly it hasn’t been updated since then, and it seemed that it was going to take a fair bit of programming to get it to do what I wanted. I then looked at various scripts to use Org-Mode files (which I use every day at work) to publish my blog; I tried three of them, but none of them worked: two would not compile and the third one failed later on. So, I had to search anew. I turned up Jekyll and Hugo. I goofed around with both, and I concluded both would require similar levels of effort, but on the list of languages I want to learn, Go is ahead of Ruby, so I went with Hugo.
So here we are. The site is missing a lot of things, but I’ll gradually work on adding them. If you have any experience with Hugo or any other advice to share, let me know.
There’s an old saying that I first heard a couple of decades ago:
Good. Fast. Cheap. Pick two.
I was listening to an NPR story about Moore’s Law. At first I was thinking that it gave us computers that are “fast” and “cheap”. But we never got the “fast”. The workstation on my desk 25 years ago was just as fast as the one I’m sitting at now. Ah, but that old Sun 3/50 didn’t have to do nearly as much as my current workstation, which is true. That old workstation didn’t have color, virtual desktops, animated 3d icons, streaming audio and video, bloated web and email programs, etc. But somehow I got my work done just as quickly. What’s happening here is another law is cancelling out the “fast” part of Moore’s Law: Wirth’s Law. That law basically says that software is getting slower faster than hardware is getting faster.
Case in point: 25 years ago, when I fired up Emacs (which served as my text editor, mail and usenet reader, and web browser), the 4 megs of virtual memory it used had a noticeable impact on other users. Nowadays Emacs is a lightweight. Right now my email client (Thunderbird) is taking up 1.2 gigs of virtual memory!
Moore, himself, acknowledged that his law has its limits, and some people place that 10 years in the future. Thus far, Moore’s Law has managed to just barely keep up with Wirth’s law. So what happens when the latter tops out? I seriously doubt the latter has any limits, as I have yet to see a limit on human wastefulness and incompetence.
I guess we’ll need to go back to the trinity listed above. Maybe we need to start doing something toward “good”: stop adding new bells and whistles and go back and fix bugs, make software more reliable and more informative when something does fail, and generally reduce all the frustration that everyone feels when using computers. In other words, do the opposite of what we’ve been doing. It’s a massive challenge, and, by and large, unfamiliar, virgin territory.
I know this is probably another one of my utopian dreams, and will probably never happen, but it would be nice if, for once, I could encounter someone using a computer and not feel the urge to apologize on behalf of my profession.
The only good thing that spammers do for us is to give us something to laugh at. The latest case was a comment in my moderation queue which started like this:
{I have|I’ve} been {surfing|browsing} online more than {three|3|2|4} hours today, yet I never found any interesting article like yours.
It continued on for quite a while, but I’ll spare you. Clearly someone screwed up their automation. Just for grins, I wanted to see if I could write a one-liner to process this input, and, viola!
perl -pe 's(\{(.+?)\})(@a=split(/\|/, $1); $a[rand(1+$#a)])ge' < spam
I’ve replaced a spammer with a one liner! If only that meant they’d go away.
I spotted this today in a discussion about Subversion, and a workaround for a situation which ended up corrupting the workspace:
Arr… the reason I’m trying to get us to switch to Git. Less of this funny business.
I will admit, I’m not a big fan of Git. But my biggest problem with it is the born-again fervor of some of its fans like the one above. I will freely admit that Git has some advantages over Subversion (but there are disadvantages as well). But to claim that workspace corruption and the attendant workarounds is something Git (or any version control system) is immune to is an indicator that someone drank too much kool-aid.
Maybe it’s just me, but I’ve been working with computers so long that I believe that every piece of software that has ever existed (or ever will exist) has its share of “funny business”. But salesmen and evangelists could never admit such a thing.
The original issue I ran into had to do with the dysfunctional practice of checking in enormous binary files. Every version control system is going to have issues with this, though to varying degrees. In the course of researching the issue, I found this passage about Git:
The primary reason git can’t handle huge files is that it runs them through
xdelta, which generally means it tries to load the entire contents of a file into memory at once. If it didn’t do this, it would have to store the entire contents of every single revision of every single file, even if you only changed a few bytes of that file. That would be a terribly inefficient use of disk space, and _git is well known for its amazingly efficient repository format.
I was with him up until that last phrase. We were having a serious technical discussion, and suddenly a salesman crashed the party! This “amazingly efficient” repository format is largely thanks to xdelta. The salesman neglected to mention that xdelta is the same mechanism used by Subversion. We could certainly have a serious, quantitative, technical discussion about the tradeoffs of various mechanisms for storing versioned data, or about the ways to manage those deltas. But something tells me that the salesmen and evangelists will crash that party as well.
That last phrase could have been more accurate and less obnoxious had it been phrased “and any modern version control system worth using would not do so.”
Here’s an error message I first remember seeing a few decades ago. Sadly I just ran into it again. It seems guessing games never go out of style.
Error while changing the NIS password.
The NIS password has not been changed on somenisserver.
For many years I was forced to use Windows on my work computers. I almost always listen to music as I work, so I needed a music player. I quickly grew dissatisfied with Windows Media Player, and after some searching I found Foobar2000, which I really liked. A utilitarian, but capable, interface which did most everything I needed and did it pretty well.
A few years ago I switched my desktop machine to Linux. Hurrah! But now the music player again became an issue. I tried Rhythmbox, Banshee and Amarok. None of them were very good. Nothing close to Foobar2000. All seemed to suffer from the same issues: lots of pointless graphical flourish, confusing and/or dumbed down interfaces, poor functionality, and lousy error handling. By process of elimination, I ended up with Banshee. A grudging choice which I regret on most days.
A main source of pain is the Podcast extension. It suffers from my usual pet peeves: no indication what, if anything, is happening and poor to non-existent error reporting. Here’s a prime example: I hit the button to check for new podcasts. Nothing seems to happen for a few minutes, but then some new episodes pop in. But they don’t download. I right click on a new episode, and select download. Nothing. I wait. Still nothing. So I go digging. After several minutes of searching I find a log file buried under my home directory, it contains this:
[Warn 09:24:14.993] HttpDownloadTask The Permaculture Podcast - Honeybees with Dr. Dennis vanEngelsdorp Error: System.Net.WebException: The remote server returned an error: (404) Not Found.
So, it actually did attempt to download, but didn’t bother to say anything. When that download failed, it quietly tucked the error into an obscure log file, and on top of that labeled it as “Warn”. I tend to be of the opinion that when I ask for something to be done, and it cannot be, that is an “Error”, not a “Warning”. But even so, the error message doesn’t mention the URL or whether it was going through a proxy, without those I still don’t know why it happened. As I recall I continued experimenting with settings until I figured out that the proxy was the issue.
I didn’t become a software engineer so I could spend my days pushing upstream against issues like this.
Years ago one of my co-workers complained that my code had “too much error handling”. I was astonished, but said little in defence since I was the new guy on the team. Looking back on this, years later, I am bothered by this attitude. It is easy to write code that works correctly when everything it depends upon works correctly. Given the complexity of modern software and hardware, there are an endless number of things which can fail.
Therefore, error handling becomes the most critical part of the code. We have to code with the assumption that anything can fail. In my experience, it will, sooner or later. When the failure does happen, it must be dealt with in a reasonable manner. Ideally that would be some sort of self-healing, retrying in the case of transient issues, and, failing that a useful and comprehensive error message.
I first started writing this post at least 4 years ago, and in the meantime it has become apparent that my point of view is the minority amongst programmers. Silent failures, incomprehensible error messages, and, crashes are a daily part of life amongst the recent wave of gadgetry. But I guess the plus side is it gives me something to complain about here.
It’s sad that I still have to try to convince people that they should be using some sort of version control. But I do. One of the things I say is that there are three kinds of things in the world: Things that are checked in, things that are generated from things that are checked in, and garbage.
I noted this error, but forgot to note where I got it. I’m pretty sure it was IBM’s support site:
There is an issue with your login that needs to be addressed before you can successfully login to SR. Please contact the SR help desk.
So I can’t log into the SR system due to an “issue”, and to get it resolved, I need to file an SR. My head is spinning, just like the logic.
I’m not sure why I didn’t think of this earlier, but I just put Defensive Omnivore Bingo onto GitHub. So if you have any contributions, feel free to send me a pull request. Of course, email still works.
Next blog post in progress…
Every day I see many of those supposed progress indicators. But they don’t indicate anything other than lazy and sloppy programming. That little icon will go on spinning regardless of what is actually happening. In the rare situations where it really does relate to work being done, it rarely relates in a useful way. It seems that these days most software is written with several assumptions in mind:
The first assumption is laughable, the second is insulting, and the third is sad.
It shouldn’t be hard to do this right. Many years ago I cooked up a simpleminded progress indicator which tells you what is happening, and about how long it might take. It was quick and stupidly simple to write.
Loading host info for urolite.example.com... 16 of 17 (94.1%) 9s ETA 1s
But as for the progress on this blog, I have nearly a dozen mostly finished drafts waiting to be posted, but I’ll leave that icon spinning so that you know that something is going on.
I often rant about lousy error messages, but it seems that a growing (and maybe more worrisome) phenomenon is silent failures. Here’s an example I just ran into with ClearCase (this is not a new bug, I first encountered it over 10 years ago):
$ ct desc -l vob:. | grep MITTR
SLC0314:C:\views\MITTR.vws [uuid 107553f5.340c4e00.bd2c.3f:52:42:42:31:7b]
$ ct rmview -force -uuid 107553f5.340c4e00.bd2c.3f:52:42:42:31:7b
Removed references to view "SLC0314:C:\views\MITTR.vws" from VOB "/vobs/factcheck".
$ ct desc -l vob:. | grep MITTR
SLC0314:C:\views\MITTR.vws [uuid 107553f5.340c4e00.bd2c.3f:52:42:42:31:7b]
What exactly is happening is unimportant, what you can see is that there was an entry in the “desc” output, then I tried to remove it, which the output indicates happened, but double-checking indicates nothing was done. It_ failed_, and lied about it.
My assertion that this is a defect was ignored by IBM Tech Support. I guess their definition of “defect” is different from the rest of us.
I could be wrong, but it seems obvious to me that this sort of failure is a classic case of someone not checking for errors, that is, leaving off the last part of something like this:
rmviewuuid($uuid, $vob) or warn "Error: failed to remove view $uuid from $vob\n";
While built-in exception handling (à la java) can help, I’ve seen plenty of code like this (even in Java textbooks):
try { vob.rmview(uuid) } catch { }
Back when I started programming, little bits of sloppiness like this weren’t a big deal. It was often not hard to see what was going wrong when a 1000 line program was running. But when you have millions of lines of code, often spread amongst numerous libraries, in many different interlocking programs, the problems grow exponentially. As we all experience daily.
Back in 1994 I was just a few years out of college, where I was often pushing people to use the new-fangled RCS system. On my first day at my new job I was handed a stack of ClearCase manuals and told help get it deployed. I was amazed. While the system was difficult to configure, it presented some amazing features, a few of which have yet to be equaled.
The Multi-Version File System was amazing. I could create a workspace in constant time. There is no system before or since (that I know of) which can claim this. The ability to change a date in a config spec and be instantly transported to what the code looked like at that time was fantastic. Being able to do a cd command (e.g. “cd foo.c@@”) and be transported into the version tree of that file seemed utterly magical all those years ago. The unequaled branching and merging abilities permitted heavily parallel development. A am still in awe of these things.
Here I am 18 years later looking at an email from IBM Tech Support where I am told that I “might be able to get this recognized as a defect” (emphasis mine). The problem was that I installed ClearCase and got no errors, but found that the installation was broken and unusable. A key component, indeed the central component, MVFS, did not get installed. In what universe is that not a defect?
But then I realized that this unfortunate tech support engineer and myself were arguing over a corpse. This product is dead, Jim. We’re like the two guys in Weekend at Bernie’s, propping up a corpse in an attempt to convince everyone he’s still alive.
ClearCase, from day one, was hobbled by several unfortunate architectural mistakes: a chatty protocol and a deep dependence on network filesystems and the attendant maze of authentication systems. Some people have said that the lack of database scalability was the key issue, but I think that actually could have been fixed. Even if it was, it wouldn’t change the fact that any amount of network latency would make ClearCase’s performance exponentially worse.
There are a variety of ways these fundamental problems could have been addressed given continuous development effort. But, I would argue that the core of ClearCase has remained unchanged for almost 15 years. The only changes made since then have been cosmetic changes, additional bolted-on bits (UCM) or replacing functional components with broken ones (like the new installer). This product has died of neglect.
In the last 5 years I have only brought one new team into ClearCase. Everyone else I’ve been steering to Subversion. For most teams that is all they need.
This is a fun one from svnsync:
Synchronizing repository some-replica...
Transmitting file data ......................................!
.......................svnsync: Error! while replaying commit
What’s even better is that a Google search reveals a number of people running into this exact error. Sadly, few of them got replies, and those that did get replies did not get helpful replies. Certainly no solutions.
Mashing the error message on the previous line is a nice touch as it makes it harder to spot the error message, especially when there are numerous periods (thousands, in my case).
I’m a firm believer that every program has to have the ability to enable debugging output, as that could compensate for such poor error messages. But in this case, no luck. Maybe the java stack trace isn’t so bad after all.
When I was a child, my parents would often tell me to repeat what they just told me, since I usually wasn’t paying attention. Now I have to do the same thing with my own daughter. Payback time, it seems.
But this blog entry isn’t about parenting, it’s about error messages.
I was just writing some code and realized that an important rule when writing error messages is to repeat back what the user said. There are many violations of this rule, the first one that comes to mind is this one from Windows:
The system cannot find the path specified.
That error may be comprehensible if you just typed a command, but as part of a script, it will be entirely useless. Obviously, the pathname needs to be displayed (of course, we still don’t know what was being done, or why).
This becomes even more important when a user specified value is modified in some way. For example I had a command line argument which could take a list. After breaking the list apart, I needed to validate the entries in the list. If I found anything invalid I could have simply given the error “invalid parameter”. Useless! Rather, I filtered out the valid values and then printed out the offending ones: “invalid parameters: a,b,c”.
Now, repeat what I just said!