How to get sandwiches on the Internet

One of our neighbors wanted to take orders online, so we hooked up their menu to post to, and then made a web page that “displayed” the orders.

The idea involved a Windows scheduled task, running on their personal computer, next to the kitchen, that would poll a web page for orders.

When new ones showed up, it would print them, and so long as the print succeeded, it would mark the orders as printed. The client code looked like this:

On Error Resume Next
Set IE = WScript.CreateObject("InternetExplorer.Application")
while IE.busy = True: WScript.Sleep 100: wend
If InStr(IE.Document.Body.innerHTML, "<!-- ok -->") &gt; 0 Then
	IE.ExecWB 6,2,2,0
	while IE.busy = True: WScript.Sleep 100: wend
	If Err.Number = 0 Then
		while IE.busy = True: WScript.Sleep 100: wend
	End If
End If

Meanwhile, the server code was a (longer, and less interesting) perl script which simply formatted the page printably (using <table> tags and everything) and had a form on it with (hidden) checkboxes. The page included an <!– ok –> right at the end to signal completedness.

Emacs as an inferior-lisp-program

Emacs has this inferior-lisp mode where you can send lisp forms to a running lisp image. This can either be a common-lisp (although Slime is more specialized at that), or a scheme, or as it turns out, it can be emacs itself.

Put these two files in a directory some place:

Then, simply tell emacs to make your inferior-lisp-program.

Before the novelty wears off, a better version might simply be a reader that speaks emacsclient…

Mike wants to keep up with you on Twitter

Anyone have any ideas why Ameriplan aka MyBenefitsPlus dot com is trawling for email addresses and searching for them on twitter?

When someone searches for your email addres on twitter and you’re not already on twitter, you get an email from twitter that looks like this:

From: Twitter <>
blah blah blah, join twitter.

From there, if you like, you can search for them; make a phone call; email them directly; etc. It all seems awful convenient, until some spammers and snake-oil vendors start trying to use it as the next “in”.

I wonder if Twitter is going to try and stop this, try and monetize it, or even let its users know that spammers are connecting their online and offline profiles through their service.

Until I learn more, I’m going to strongly recommend that people only use twitter (if at all) using secret email addresses.

fsync is cheaper than you think

With all the talk about the new ext4’s interactions with gconfd, kde, firefox, and others, a lot of people have been assuming fsync is expensive.

If you’re living under a rock and don’t know what’s going on, here’s the short version: Lots of programs don’t write to files reliably because (a) writing to files reliably requires some contortions, and (b) writing to files reliably is slow.

Neither of these things are true, and I wrote libreliable to prove it.

libreliable is a simple, mostly portable way to get reliable file i/o
without an fsync()-penalty after every write.

That means you have no excuse for not updating your files reliably.

The included demo should show how it works. It’s fun to watch in strace/truss.

Why nobody likes a PHP developer

This isn’t hate on PHP.

We do web hosting. Most of our customers use PHP in some capacity or another. PHP has this great set of features that made it very accessible for a wide array of applications. People using it means there are lots of applications available for it. Lots of people. Lots of applications.

This is a good thing. Users have lots of choice about what programs they want to use, and there is a lot of competition to produce high quality, very polished user-interfaces. Users like those things.

Perl and Python programmers should take note: The number of polished and high-quality open source PHP applications is positively staggering. I can only name a few Perl and Python ones off the top of my head (and btw, Bugzilla is great; this isn’t Perl hate either), but if not from anything but the sheer accessibility of PHP, there is a lot of great stuff out there.

However users of PHP applications must wear diapers and be idiots because I cannot find a single PHP application on freshmeat that lets the user enter in the PDO dsn directly, or otherwise directly control the arguments to mysql_connect().

I wanted to take a look at piwik recently and I found it’s impossible to install. I suppose it’s their fault, and I opened a bug there about it, to which they immediately closed the bug with the friendly response:

Thank you for your interest in Piwik. However, we can’t provide this level of MySQL support. Please try the MySQL forums.

PHP Bug #34988 is somewhat symptomatic of the problem, but not the problem itself. The proposed “solution” (note the bug is marked “Bogus”) is to put something into your DSN. Edward Sapir would be so proud. Recall that piwik, like most PHP applications, doesn’t let you enter the DSN. You’re stuck with with a brain-damaged “default” to look at whatever mysql_config –socket said on the machine that compiled PHP itself.

In this case, I would be able to work around the problem by typing in my “host name” as:


except for another bug in piwik (that thankfully has been “fixed”). Oh well.

It’s not really the piwik developers fault- PHP encourages a certain approach to things, and in at least this thing, PHP is completely and fundementally wrong. Annoyingly wrong.

The developers of PHP could fix this, of course. Add another override in php.ini perhaps. Maybe a mysql_magic_localhost= option, or similar.

Another way to go would be to encourage the desired behavior. PDO should accept a “short” DSN that is registered in the php.ini file. To support run-time database selection like phpMyAdmin, allow a “long” DSN to be specified, but only if a $_COOKIE[] is set that contains the SHA1-hash of the long DSN (plus some secret value).

If this were the case, PHP programs wouldn’t feel the need to individually, and collectively, reinvent the database-connection-wizard over and over again in subtly broken ways. I understand it’s trying to help me: asking for my mysql server name, username, and password. I’m not interested in supplying any of these because having the PHP script know these things is itself a security risk. If the PHP script can log in, then perhaps something else can… Having the PHP script have a username and password for your database server is a security risk.

Of course, it’s much too late for that. Backwards compatibility is far more important than the “right” API calls. Especially with PHP6 creaking around the corner about to remind us hosting providers why we still need to offer a PHP4 system.

For now, we’ll have to rely on advocacy, and an otherwise absurd argument: The assumption that the user is too stupid to enter a DSN in that their system administrator gives them, but the user is capable enough to ask and enact a complex set of firewall rules, is frankly stupid, and so is, by extension, your database connection wizard.

To the PHP developers using mysql_connect: Just use the defaults. Don’t ask the user for ANYTHING unless mysql_connect() with no argument fails. Even then, encourage the user to get their php.ini file correct. You don’t need any configuration, and most of the time, it’ll just work.

PHP developers using PDO: Let the user enter a DSN. It’s fine if you want to be helpful and “pre-populate it” with hostnames usernames and passwords, but the fact is you’re perpetuating a huge security risk, and there’s just no need for it. The DSN is designed to be edited by people, not you, not the programmer. If it were just for your program, it would be an array() of some sorts.

Developers of PHP: Nobody likes you because of crap like this. There’s nothing about the defaults here that make sense. The fact that they “accidentally” work for the case of servername=localhost is the only thing that lets Debian ship you. There are a dozen other ways you could do this, including naming DSNs in the php.ini file, smarter APIs, consulting the .my.cnf file that the mysql program uses, and even letting us override defaults like this at run-time. Please stop being so disappointing all the time, you’re difficult enough to support as it is.

Interactive HTML development in Emacs

I accidentally discovered mozrepl today. It’s clearly designed for JavaScript development, but I thought it might be interesting for HTML development as well, and hacked the following together:

(require 'moz)
(require 'json)
(defun moz-update (&rest ignored)
  "Update the remote mozrepl instance"
  (comint-send-string (inferior-moz-process)
    (concat "content.document.body.innerHTML="
             (json-encode (buffer-string)) ";")))
(defun moz-enable-auto-update ()
  "Automatically the remote mozrepl when this buffer changes"
  (add-hook 'after-change-functions 'moz-update t t))
(defun moz-disable-auto-update ()
  "Disable automatic mozrepl updates"
  (remove-hook 'after-change-functions 'moz-update t))

With this, Firefox will update its HTML contents as you type, for whatever buffer you’ve called moz-enable-auto-update on.

It took about five minutes to write, and another five to comment and put up here. I’ve been using it for a few hours now and haven’t found any real problems.

Edit: You’ll already need mozrepl installed, and the mozrepl emacs integration setup..

Edit: Fixed link. Thanks.

XEN musings

When putting together our VPS system, we spent a lot of time looking at XEN. Most VPS providers use XEN. XEN has great tools.

However, we eventually settled on using KVM because it was conceptually simpler, had better I/O performance (according to hdparm and ab), but most importantly because XEN’s future began to look bleak. XEN might have better tools now, we figured, but KVM will get better attention.

Most of our VPS customers so far have moved from XEN-based solutions, and greatly appreciate the performance difference. I wanted to note some of the things I’ve noticed helping them move from their current providers:

  • XEN hypervisor’d kernels don’t boot under KVM (hint: rsync –exclude /boot)
  • Installs are a lot slower on KVM than on XEN (perhaps it’s a problem with CDROM emulation)
  • fsync performance is lower on KVM and XEN; noatime helps a lot
  • Lots of providers seem to create swap as a file inside the guest. This appears to be done to simplify accounting, but we get enormous performance gains by not doing this (even under XEN)
  • CPU tasks are a lot faster under XEN. If you’re compiling things all day, XEN (or a regular virtual hosting account) might make more sense.

The KVM Forum reported lots of really interesting developments that our VPS customers will be able to take advantage of, that XEN-based VPS users will simply miss out on, at least until xenner becomes stable.

Network Maintenance

The following servers will experience temporary downtime while maintenance occurs in the NOC.

  • Hermes
  • Starscream
  • Canis
  • Bacchus
  • Icarus
  • Tantalus
  • Diana
  • Leto
  • IDSA
  • Nyx
  • YMEweb
  • Ampolweb
  • Echo
  • Golconda
  • Artebe

If your account rests on one of these servers, you will be notified in advance via email as to the exact date and time of your Maintenance.

Basically, we are cleaning house. In the past year we have had some significant changes in our network. Our new ISP has installed an OC-12 Fiber optic rack and with all of that new bandwidth we were able to close out one of our smaller Datacenters and move them back home. The change has done everyone, especially our clients, alot of good. The only downside to all that rapid growth is mess, and we made a BIG one.

The goal of this downtime is to organize our cabling, install a new KVM switch for racks 20,21,and22, and get ourselves ready for the next wave of internal changes.


As you can see… it’s pretty messy in there. However, we are going to take one of these…


And hook it up to all of these…


And tie that whole section together to make our lives ALOT easier, and to make our next batch of upgrades as seamless as possible.

Security by what?

One of our customers called and said he couldn’t connect to netMail, and after some digging, I found out that the problem only occurred on the inside.

A Sonicwall firewall was to blame; butchering POST content that produces application/x-javascript – but not the GET content with the same mime type, and not when the content is decorated with text/plain

Being as how <script> ignores the MIME type in every major browser, I cannot see why anyone would think this is a good idea, and yet using a separate MIME type just for the RPC makes using wireshark easier.

If you use a Sonicwall, netMail will now work with your firewall, but you have to wonder what kind of security vendor thinks this added complexity actually buys anything…

Backing up Windows Servers

I had always assumed that NTBACKUP.EXE did the right thing. Somehow, magically, it grabbed all the files – even those that were open. However, in upgrading our backup systems, I noticed why most recovery requests for Windows sites fail: Windows cannot back up files that are open by other processes. This means that the ultra-important MDB file you’re using as a database cannot be saved unless your application saves it someplace.

Well, nobody does that, so I decided to do something about it.

I took NTFS-3G- the NTFS driver most often used by Linux users- and PuTTY- a popular SSH client, and glued them together to produce WINBACK.EXE, backup software for windows that actually works:

Basically, if the BackupRead api fails because the “file is in use”, WINBACK.EXE uses NTFS-3G to read the raw disk sectors containing the file. It then uploads these files to our backup server, and if it accepts it, removes the archive-bit.

I only found one other software package that does it. It’s a kernel driver which means it can muck up your ability to do restores. Nevertheless, if your backup software doesn’t use Softpedia’s FAM, and it isn’t my WINBACK.EXE, then it simply doesn’t work.

WINBACK.EXE took less than a week to hack out, so it could use some polish. Nevertheless, it does a much better job than anything else, and because it’s basically a copy-utility, recovery is much easier than with other systems.

By the way, I am aware of Microsoft’s Volume Shadow Copy Service. It doesn’t work on Windows 2000, and it doesn’t work well on Windows 2003.