March 24, 2010Hello from Montreal! I’m writing this from a wireless connection up on the thirty-ninth floor of La Cité. Unfortunately, when we reading the lease, the only thing we checked was that it had “Internet”… not “Wireless.” So what’s a troop of MIT students with an arsenal of laptops and no wireless router to do? Set up wireless ad hoc networking.
Except it doesn’t actually work. Mostly. It took us a bit of fiddling and attempts on multiple laptops to finally find a configuration that worked. First, the ones that didn’t work:
- Windows, as Daniel Gray tells me, has two standard methods for creating ad hoc networks: bridging two networks or …. We tried both of them, and with … we were able to connect other Windows laptops and Mac OS X laptops… but no luck with the Linux laptops. As three of us are Linux users, we were quite unhappy with this state of affairs.
- Linux theoretically has support for ad hoc networks using dnsmasq; however, we tried two separate laptops and neither of them were able to set up an ad hoc network that any of the other laptops were able to use. We did discover some hilarious uninitialized field bugs for ESSIDs.
- Mac OS X. At this point, we were seriously considering going out, finding a wireless hardware store, and buying a router for the apartment. However, someone realized that there was one operating system we hadn’t tried yet. A few minutes of fiddling… and yes! Ad hoc network that worked on all three operating systems!
Ending score: Apple +1, Microsoft 0, Linux -1. Although, it’s hard to be surprised that no one actually is paying the attention necessary to the wireless drivers.
March 22, 2010Abstraction (n.) The act or process of separating in thought, of considering a thing independently of its associations; or a substance independently of its attributes; or an attribute or quality independently of the substance to which it belongs. (Oxford English Dictionary)
Abstraction is one of the most powerful beasts in the landscape of programming, but it is also one of the most elusive to capture. The places where abstraction may be found are many:
- Good artists copy. Great artists steal. Abstractions that other people have (re)discovered, (re)used and (re)implemented are by far the easiest to find.
- First time you do something, just do it. Second time, wince at duplication. Third time, you refactor. Refactoring introduces small pieces of ad hoc abstraction. The quality varies though: the result might be something deeper, but it might just be mundane code reuse.
- Grow your framework. A lengthy process where you build the abstraction and the application, build another distinct application on the abstraction, and reconcile. This takes a lot of time, and is entirely dependent on people’s willingness to break BC and make sweeping changes.
- Give the problem to a really smart person. Design is a creative process, design by committee results in moldy code. A single person unifies the abstraction and picks the battles to fight when an abstraction needs to change.
- Turn to nature. User interfaces often introduce a form of abstraction, and it’s common to turn to real life and see what’s there. Note that there are really good abstractions that don’t exist in real-life: the concept of undo is positively ridiculous in the real world, but probably one of the best inventions in computers.
I’d like to propose one other place to turn when you’re hunting for abstractions: pure mathematics.
“Pure mathematics?”
Yes, pure mathematics. Not applied mathematics, which easily finds its place in a programmers toolbox for tackling specific classes of problems.
“Ok, mathematicians may do neat things… but they’re too theoretical for my taste.”
But that’s precisely what we’re looking for! Pure mathematics is all about manipulating abstract objects and deductively proving properties about them. The mathematicians aren’t talking about a different kind of abstraction, they’re just starting off from different concrete objects than a programmer might be. The mathematicians have just have been doing it much longer than programmers (set the mark at about 600BC with the Greeks.) Over this period of time, mathematicians have gotten pretty damn good at creating abstractions, handling abstractions, deducing properties of abstractions, finding relationships between abstractions, and so forth. In fact, they’re so good that advanced mathematics has a reputation for abstracting concepts way beyond what any “normal” person would tolerate: Lewis Carroll is one prominent figure known for satirizing what he saw as ridiculous ideas in mid-19th century mathematics.
Of course, you can’t take an arbitrary mathematical concept and attempt to shoehorn it into your favorite programming language. The very first step is looking at some abstract object and looking for concrete instances of that object that programmers care about. Even structures that have obvious concrete instances have more subtle instances that are just as useful.
It is also the case that many powerful mathematical abstractions are also unintuitive. Programmers naturally shy away from unintuitive ideas: it is an example of being overly clever. The question of what is intuitive has shaped discussions in mathematics as well: when computability theory was being developed, there were many competing models of computation vying for computer scientists’ attentions. Alonzo Church formulated lambda calculus, a highly symbolic and abstract notion of computation; Alan Turing formulated Turing machines, a very physical and concrete notion of computation. In pedagogy, Turing machines won out: open any introductory computability textbook (in my case, Sipser’s textbook) and you will only see the Turing machines treated to the classic proofs regarding the undecidability of halting problems. The Turing machine is much simpler to understand; it maps more cleanly to the mental model of a mathematician furiously scribbling away a computation.
But no computer scientist (and certainly not Alonzo Church) would claim that Turing machines are the only useful model to study. Lambda calculus is elegant; after you’ve wrapped your head around it, you can express ideas and operations concisely which, with Turing machines, would have involved mucking around with encodings and sweeping the head around and a lot of bookkeeping. Writing Turing machines, put bluntly, is a pain in the ass.
I now present two examples of good ideas in mathematics resulting in good ideas in programming.
The first involves the genesis of Lisp. As Sussman tells me, Lisp was originally created so that McCarthy could prove Gödel’s incompleteness theorem without having to resort to number theory. (McCarthy is quoted as saying something a little weaker: Lisp was a “way of describing computable functions much neater than the Turing machines or the general recursive definitions used in recursive function theory.”) Instead of taking the statement “this statement cannot be proven” and rigorously encoding in a stiff mathematical formalism, it could simply be described in Lisp (PDF). Thus, its original formulation featured m-expressions, which looked a little like function[arg1 arg2] and represented the actual machine, as opposed to the s-expression which was the symbolic representation. It wasn’t until later that a few graduate students thought, “Hm, this would actually be a useful programming language,” and set about to actually implement Lisp. Through Gödel’s incompleteness theorem, the powerful notion of code as data was born: no other language at the time had been thinking about programs in this way.
The second is the success of Category Theory in Haskell. The canonical example is monads, the mathematical innovation that made input/output in lazy languages not suck (although a mathematical friend of mine tells me monads are not actually interesting because they’re not general enough.) But the ideas behind the functor and applicative functor encapsulate patterns pervasive in all programming languages. An example of a reformulation of this concept can be seen in numpy’s universal functions. They don’t call it a functor, instead they use terms such as “broadcasting” and “casting” and discuss the need to use special universal versions of functions on numpy arrays to get element-by-element operations. The interface is usable enough, but it lacks the simplicity, elegance and consistency that you get from actually realizing, “Hey, that’s just a functor…”
Those mathematicians, they’re smart folk. Perhaps we programmers could learn a thing or two from them.
Postscript. Thanks to Daniel Kane for answering my impromptu question “What is mathematics about?” and suggesting a few of the examples of mathematics leaking back into computer engineering.
March 19, 2010Python is a language that gives you a lot of rope, in particular any particular encapsulation scheme is only weakly enforced and can be worked around by a sufficiently savvy hacker. I fall into the “my compiler should stop me from doing stupid things” camp, but I’ll certainly say, dynamic capabilities sure are convenient. But here’s the rub: the language must show you where you have done something stupid.
In this case, we’d like to see when you have improperly gone and mutated some internal state. You might scoff and say, “well, I know when I change my state”, but this is certainly not the case when you’re debugging an interaction between two third party libraries that you did not write. Specifically I should be able to point at a variable (it might be a local variable, a global variable, or a class/instance attribute) and say to Python, “tell me when this variable changes.” When the variable changes, Python should tell me who changed the variable (via a backtrace) and what the variable changed to. I should be able to say, “tell me when this variable changed to this value.”
Well, here is a small module that does just that: mutsleuth. Import this module and install the watcher by passing mutsleuth.watch an expression that evaluates to the variable you’d like to check.
Here’s an example: suppose I have the following files:
good.py:
b = "default value"
evil.py:
import good
good.b = "monkey patch monkey patch ha ha ha"
test.py:
import mutsleuth
mutsleuth.watch("good.b")
import good
import evil
When you run test.py, you’ll get the following trace:
ezyang@javelin:~/Dev/mutsleuth$ python test.py
Initialized by:
File "test.py", line 5, in <module>
import evil
File "/home/ezyang/Dev/mutsleuth/good.py", line 1, in <module>
b = "good default value"
Replaced by:
File "test.py", line 5, in <module>
import evil
File "/home/ezyang/Dev/mutsleuth/evil.py", line 2, in <module>
good.b = "monkey patch monkey patch ha ha ha"
There are a few caveats:
- Tracing doesn’t start until you enter another local scope, whether by calling a function or importing a module. For most larger applications, you will invariably get this scope, but for one-off scripts this may not be the case.
- In order to keep performance tolerable, we only do a shallow comparison between instances, so you’ll need to specifically zoom in on a value to get real mutation information about it.
Bug reports, suggestions and improvements appreciated! I went and tested this by digging up an old bug that I would have loved to have had this module for (it involved logging code being initialized twice by two different sites) and verified it worked, but I haven’t tested it “cold” yet.
Hat tip to Bengt Richter for suggesting this tracing originally.
March 17, 2010Last week I talked about how we replaced a small C program with an equivalent piece of Haskell code. As much as I’d like to say that we deployed the code and there was much rejoicing and client side caching, the real story is a little more complicated than that. There were some really good questions that we had to consider:
How many maintainers at any given time know the language? The Scripts project is student-run, and has an unusually high turnover rate: any given maintainer is only guaranteed to be around for four to five years (maybe a little longer if they stick around town, but besides a few notable exceptions, most people move on after their time as a student). This means at any given point we have to worry about whether or not the sum knowledge of the active contributors is enough to cover all facets of the system, and facility in a language is critical to being able to administrate the component effectively (students we are, we frequently don both the sysadmin and developer hats). In a corporate setting, this is less prominent, but it still plays a factor: employees switch from one group to another and eventually people leave or retire. We have two current maintainers who are fairly fluent in Haskell. The long-term sustainability of this approach is uncertain, and hinges on our ability to attract prospective students who know or are interested in learning Haskell; in the worst case, people may crack open the code, say “what the fuck is this” and rewrite it in another language.
How many maintainers at any given time feel comfortable hacking in the language? While superficially similar to the first point, it’s actually quite different; posed differently, it’s the difference between “can I write a full program in this language” and “can I effectively make changes to a program written in this language.” At a certain level of fluency, a programmer picks up a special feat: the ability to look at any C/Fortran derived language and lift any knowledge they need about the syntax from the surrounding code. It’s the difference between learning syntax, and learning a new programming paradigm. We may not be simultaneously Python/Perl/PHP/Java/Ruby/C experts, but the lessons in these languages carry over to one another, and many of us have working “hacker” knowledge in all of them. But Haskell is different: it’s lineage is among that of Lisp, Miranda and ML, and the imperative knowledge simply does not translate. One hopes that it’s still possible to tell what any given chunk of Haskell code does, but it’s a strictly read-only capability.
Who else uses it? For one of the team members, migrating from Subversion to Git was a pretty hard sell, but at this point, minus the missing infrastructure for doing the migration properly, he’s basically been convinced that this is the right way forward. One of the big reasons this was ok, though, was because they were able to list of projects (Linux, our kernel; AFS, our filesystem; Fedora, our distro) that they used regularly that also used Git. We can’t say the same for Haskell: the “big” open-source high-visibility applications in Haskell are Xmonad and Darcs, of which many people have never used. As a student group, we have far more latitude to experiment with new technology, but lack of ubiquity means greater risk, and corporations are allergic to that kind of risk.
Is the ecosystem mature? Internally, we’ve given the Ruby maintainers and packagers a lot of flak for a terrible record at backwards compatibility (one instance left us unable to globally update our Rails instances because the code would automatically break the site if it detected a version mismatch). You see a little bit of the same in Haskell: static-cat doesn’t actually build on a stock Fedora 11 server with the default packages installed, due to an old version of the cgi module that uses the Exception backwards compatibility wrapper and thus is incompatible with the rest of the exception handling code in the program. Further investigation reveals that the cgi module is not actually being actively maintained, and the Fedora cabal2spec script is buggy. I’ve personally had experiences of coming back to some Haskell code with up-to-date libraries from Hackage and finding that API drift has made my code not compile anymore. Cabal install refuses to upgrade all of your packages in one go.
There are many ways to work around this. A mitigating factor is that once you’ve compiled a Haskell program, you don’t have to worry about package composition anymore. Workarounds include rewriting our code to be forwards and backwards compatible, doing stupid Fedora packaging tricks to make both versions of cgi live on our servers, convincing upstream that they really want to take the new version, or maintaining a separate system wide cabal install. But it’s not ideal, and it makes people wonder.
I’m quite blessed to be working in an environment where the first point is really the point. Can we introduce Haskell into the codebase and expect to be able to maintain it in the long run? There’ll always be C hackers on the team (or at least, there better be; some of our most important security properties are wrapped up in a patch to a kernel module), but will there always be Haskell hackers on the team? There’s no way to really know the answer to the question.
I personally remain optimistic. It’s an experiment, and you’re not going to get any better chance to make this happen than in this environment. The presence of Haskell code may attract contributors to the project that may not have been originally drawn by the fact that, down beneath it all, we’re a “gratis shared web hosting provider” for our community. Haskell seems singularly aligned to be the language to break into mainstream (sorry Simon!) And when was there ever any innovation without a little risk?
March 15, 2010The importance of constraint is one well known to those who embark on creative endeavors. Tell someone, “you can do anything you want: anything at all,” and they will blank, paralyzed by the infinite possibility. Artists welcome constraint. Writers like the constraint of a sonnet because it imposes form and gives a place to start; roleplaying groups like the constraint of a campaign setting because it imposes rules and sets the scene for the story to be told; jazz musicians like the constraint of the chords underlying an improvisation because it keeps the soloist anchored to the source tune and suggests ideas for the melody.
However, many programmers don’t the like the constraint of a type system. “The static type system doesn’t let me do what I want to.” “I needed to write four classes for what would have been two lines of Python!” “What? I can’t do that? Why not?” For them, it’s like a straightjacket. How does anyone ever get anything done when constraint ties you up?
I beg to differ. Accept the straightjacket. The things it will let you do… are surprising.
The straitjacket was historically used as an implement to prevent dangerous individuals from harming themselves and others. Programmers are not quite mental asylum inmates, though at a glance it may seem that we’ve been trying to reduce the ways for us to hurt ourselves. But such changes have often brought with them benefits, and many have eagerly traded away pointers and manual memory management for increased expressiveness.
Static types, however, are still a pain point for many people, and Haskell is an unusually constrained language due to its type system. An overenthusiastic user of Haskell’s type system might exclaim, “after I made it typecheck, it just worked!” Of course, this statement is not actually true; there is a certain essential complexity to classes of algorithms that mean the type system won’t catch the fact that you seeded your hash function with the wrong magic number.
But not all code is like this. A lot of code is just plain boring. It’s the code that generates your website, or logs your errors; it’s the code that serves as the glue for your build infrastructure, or it shuffles data from a file into an in-memory representation into a database. It’s the code is foundational; it is the code that lets you express simple ideas simply. When you look at the development of this code, the errors being made are very simple mental typos, they’re the ones that take a total of fifteen seconds to track down and fix once they manifest, but if rolled up in the time it takes to run your test suite or, dare I say it, manually test, quickly ticks to the minutes. A fast static type checker saves you so much pain, whether or not it is a Haskell compiler or pylint -e. The difference is that pylint -e is optional; there is no guarantee that any given Python project will play nicely with it, and it is frequently wrong. The Haskell compiler is not.
This is a specific manifestation of a more general phenomenon: types reduce the number of ways things can go wrong. This applies for complicated code too; (a -> r) -> r may not illuminate the meaning of the continuation to you, but it certainly puts a lot of restrictions on how you might go about implementing them. This makes it possible to look at the types without any understanding of what they mean, and mechanically derive half of the solution you’re looking for.
This is precisely how types increase expressiveness: it’s really hard for people to understand dense, highly abstracted code. Types prevent us from wading too far off into the weeds and make handling even more powerful forms of abstractions feasible. You wouldn’t rely on this in Python (don’t write Haskell in Python!), and in the few cases I’ve written higher-order functions in this language, I’ve been sure to also supply Haskell style type signatures. As Simon Peyton Jones has said, the type offers a “crisp” succinct definition of what a function does.
Even more striking is Haskell’s solution to the null pointer problem. The exception that strikes terror in the hearts of the Java programmer is the NullPointerException: it’s a runtime exception, which means that it doesn’t need to be explicitly declared in the throws specification of a method; a testament to the fact that basically any dereference could trigger this exception. Even in Java, a language of static typing, the type system fails to encode so basic a fact as “am I guaranteed to get a value here?”
Haskell’s answer to this problem is the Maybe type, which explicitly states in the type of a function that the value could be Nothing (null) or Just a (the value). Programmers are forced to recognize that there might not be anything, and explicitly handle the failure case (with maybe) or ignore it (with fromJust, perhaps more appropriately named unsafeFromJust). There’s nothing really special about the data type itself; I could have written a Java generic that had the same form. The key is the higher order functions that come along with the Functor, Applicative, Monad, MonadPlus, Monoid and other instances of this type. I’d run straight into a wall if I wanted to write this in Java:
pureOperation <$> maybeVal
<$>, a higher order function also known as fmap, is critical to this piece of code. The equivalent Java would have to unpack the value from the generic, perform the operation on it, and the pack it up again (with conditionals for the case that it was empty). I could add a method that implements this to the Maybe interface, but then I wouldn’t have an elegant way of passing pureOperation to these method without using anonymous classes… and you’ve quickly just exploded into several (long) lines of Java. It becomes dreadfully obvious why the designers didn’t opt for this approach: an already verbose language would get even more verbose. Other languages aren’t quite as bad, but they just don’t get close to the conciseness that a language that celebrates higher order operators can give you.
In summary, while it may seem odd to say this about a language that has (perhaps undeservedly) earned a reputation for being hard to understand, but the constraint of Haskell’s type system increases the tolerance of both writer and reader for abstraction that ultimately increases expressiveness. Problems that people just shrugged and claimed, “if you want to fix that, you’ll have to add tons of boilerplate,” suddenly become tractable. That’s powerful.
One final note for the escape artists out there: if you need the dynamic typing (and I won’t claim that there aren’t times when it is necessary), you can wriggle out of the static type system completely! Just do it with caution, and not by default.
March 12, 2010When I was seventeen, I wrote my very first shell script. It was a Windows batch file, bits and pieces very carefully cargo-culted from various code samples on the web. I had already had the exquisite pleasure of futzing with pear.bat, and the thought of scripting was not something I relished; “why not write the damn thing in a real programming language!” (The extra delicious bit was “a real programming language” was PHP. Hee.)
Eventually I came around to an all-Unix environment, and with it I began to use bash extensively. And suddenly, shell scripting made a lot more sense: you’ve been writing the damn commands day in and day out, just write them to a script instead! There was, however, still the pesky little problem that shell scripts are forever; like it or not, they’ve become pieces of maintained code. Entire build infrastructures have been built on top of shell scripts. They breed like rabbits; you have to be careful about the little buggers.
Here are five tips and tricks to keep in mind when tossing commands into a shell script that will make maintenance in the long-run much more pleasant!
Learn and love to use set. There is almost always no good reason not to use the -e flag, which causes your script to error out if any command returns with a nonzero exit code, and -x can save you hours of debugging by printing precisely what command the script is executing before executing it. With the two enabled, you get very simple “assertions” in your shell script:
check_some_condition
! [ -s "$1" ]
although, if at all possible, you should write error messages to accompany them.
Just because you don’t define subprocedures when you’re at your terminal (or do you? see alias and friends) and use reverse command history search with C-r doesn’t mean it’s acceptable to repeat commands over and over again your shell script. In particular, if you have a set of commands that might go into a separate script, but you feel funny about making a separate file, stuff them in a subprocedure like this:
subcommand() {
do_something_with "$1" "$2"
}
In particular, argument passing acts exactly the same way it does in a real shell script, and generally you can treat the subcommand as if it were it’s own script; standard input and output work the way you expect them to. The only differences is are that exit exits the whole script, so if you’d like to break out of a command use return instead.
Argument quoting in shell scripts is a strange and arcane domain of knowledge (although it doesn’t have to be; check out Waldman’s notes on shell quoting). The short version is you always want to wrap variables that will be interpolated with quotes, unless you actually want multiple arguments semantics. I have mixed feelings about whether or not literals should be quoted, and of late have fallen to the dismal habit of not quoting them.
Believe it or not, shell scripting has functional programming leanings. xargs, for example, is the quintessential “map” functionality. However, if the command you are pushing arguments to doesn’t take multiple arguments, you can use this trick:
pgrep bash | while read name; do
echo "PID: $name"
done
Shell scripting feels incredibly natural when speaking imperatively, and mostly remains this way when you impose control flow. However, it is absolutely a terrible language for any data processing (exhibit 1: sed and perl pipelines) and you should avoid doing too much data crunching in it. Creating utility scripts in more reasonable languages can go a long way to keeping your shell scripts pretty.
March 10, 2010C is the classic go-to tool for small programs that need to be really fast. When scripts.mit.edu needed a small program to be a glorified cat that also added useful HTTP headers to the beginning of its output, there was no question about it: it would be written in C, and it would be fast; the speed of our static content serving depended on it! (The grotty technical details: our webserver is based off of a networked filesystem, and we wanted to avoid giving Apache too many credentials in case it got compromised. Thus, we patched our kernel to enforce an extra stipulation that you must be running as some user id in order to read those files off the filesystem. Apache runs as it’s own user, so we need another small program to act as the go-between.)
It’s also a frankenscript, a program that grew out of the very specific needs of our project that you will not find anywhere else in the world. As such, it’s critically important that the program is concise and well-defined; both properties that are quite hard to get in C code. And it only gets worse when you want to add features. There were a number of small features (last modified by headers, byte ranges) as well as a number of large features (FastCGI support). None of the development team was relishing the thought of doubling the size of the C file to add all of these enhancements, and rewriting the program in a scripting language would cause a performance hit. Benchmarks of replacing the script with a Perl CGI made the script ten times slower (this translates into four times slower when doing an end-to-end Apache test).
But there is another way! Anders writes:
So I had this realization: replacing it with a compiled Haskell CGI script would probably let us keep the same performance. Plus it would be easy to port to FastCGI since Haskell’s FastCGI library has the same interface.
And a few weeks later, voila: static-cat in Haskell. We then saw the following benchmarks:
$ ab -n 100 http://andersk.scripts.mit.edu/static-cat.cgi/hello/hello.html
Requests per second: 15.68 [#/sec] (mean)
$ ab -n 100 http://andersk.scripts.mit.edu/static-cat.perl.cgi/hello/hello.html
Requests per second: 7.50 [#/sec] (mean)
$ ab -n 100 http://andersk.scripts.mit.edu/static-cat.c.cgi/hello/hello.html
Requests per second: 16.59 [#/sec] (mean)
Microbenchmarking reveals a 4ms difference without Apache, which Anders suspects is due to the size of the Haskell executable. There is certainly some performance snooping to be done, but the Haskell version is more than twice as fast as the Perl version on the end-to-end test.
More generally, the class of languages (Haskell is just one of a few) that compile into native code seem to be becoming more and more attractive replacements for tight C programs with high performance requirements. This is quite exciting, although it hinges on whether or not you can convince your development team that introducing Haskell to the mix of languages you use is a good idea. More on this in another blog post.
March 8, 2010It’s a sunny day in your advanced symbolic programming class. Your teacher has just started going over monads—in Scheme, though—and you sit in the back of the classroom snarking about little tidbits of knowledge you know from Haskell. Suddenly, the teacher says (quite earnestly too), “Edward here seems to know a lot about monads. Why don’t we have him come up and teach them to the class?” Suddenly, you’re up expounding types to people who have never used Haskell before and failing utterly to explain to people how the continuation monad works. Only after several iterations do you manage to partially rewrite the presentation in a form that doesn’t assume fluency in Haskell. You’ve fallen into the expert trap.
You’re an expert. You are in possession of in-depth knowledge, have accumulated wisdom and intuition, and all-in-all can work much more effectively than others within your domain. You might have an ego; you might get into hot arguments with other experts. Or you might be very unassuming and thoughtful; your expertise has little to do with your ego.
But unless you’ve been paying attention to the pre-requisite knowledge you assume, you will be terrible at teaching your area of expertise. Your expertise is getting in the way of teaching effectively, for the expert assumes too much prerequisite knowledge.
What do I mean when I speak of prerequisite knowledge? I don’t mean prerequisite “facts”—what is an iterative algorithm to solve linear equations, how does one reverse a list using a fold, how do X in my favorite framework. I do mean foundational knowledge: abstractions and higher-order primitives to think with—linear algebra, reducing higher-order operators and the architecture of said framework. One answers “how.” Another answers “Why.”
All of engineering and mathematics is perpetually in search of the right abstraction to tackle a problem. Perhaps the most striking change that occurs when you’ve put the problem in the right representation is that it becomes substantially shorter and easier to manipulate at a higher level. It’s no surprise that Newton needed to invent Calculus in order to develop his ideas about physics. The high-level programming languages and systems we build today would have been inconceivable in pure assembly language or silicon.
Finding and understanding the right abstraction is enlightenment: it makes hard things easy and impossible things possible. Calculations that used to take a page now are succinctly described in a sentence. The structure of the verbose system is encoded into the abstraction, leaving behind the salient pieces of the problem. Much of the same could be said for programs: before the advent of high level languages, assembly programs could fit on a few pages and be understood by a single programmer. They had to be. Modern software has gone far beyond.
In both cases, an expert will look at this new formulation, and immediately understand. The beginner, perhaps familiar with but not proficient in this encoding, has to now work out the underlying foundation again (or risk stumbling around with a simpler but faulty set of premises).
You might say, “Well, that’s not the problem of the expert; they just don’t have the prerequisites! I will teach them this topic once they learn that foundation.” This is not acceptable. It is true that formal education can grant them a familiarity with the basic primitives and relations of the abstraction; it is especially effective at weeding out false conceptions. But the facility that an expert has for an abstraction only comes when you spend some time “in the trenches”, using and applying the abstraction to bigger problems.
You might say, “I am not that uncharitable; I’ll teach the prerequisites too.” You might even expect to be able to impart knowledge upon the listener! Undelude yourself. In all but the simple topics (the ones where the simple statement of the solution is enough to illuminate), they won’t understand if you simply lecture to them. The teaching is just a roadmap for doing, the only way to truly get a visceral feel for any hard problem.
What you should say is, “I am just one lantern in the novice’s arsenal of understanding. I seek to illuminate precisely what the novice doesn’t think to look at.” In fact, there is an easy way to fulfill this purpose: force the novice to teach! They will start off with a very limited and ill-defined mental model of the concept: of the many roads to understanding, there is only one that they know. They will explain it in brutal detail of all their missteps and your implicit knowledge. They will be asked questions, and those questions will force them to clarify their understanding of this path. Eventually they will feel confident in their knowledge of the path, and if they continue learning, that path will expand to encompass many paths, different routes to understanding. The novice has become an expert. But, as the Buddha might say, they are the ones who must discover enlightenment. The teacher merely shows them the path.
March 5, 2010There are lots of little blog posts containing advice about various one-line options you can do in Vim. This post falls into that category, but I’m hoping to do a more comprehensive view into one small subsystem of Vim’s configuration: automatic line wrapping.
When programming, automatic line wrapping can be a little obnoxious because even if a piece of code is hanging past the recommended 72/80 column width line, you probably don’t want to immediately break it; but if you’re writing a text document or an email message, that is specifically the behavior you want. By default, vim does no automatic line wrapping for you; turning it on is a question of being able to toggle it on and off when you want it.
Here are the configuration options you care about:
- textwidth (or tw): controls the wrap width you would like to use. Use
:set tw=72 to set the wrap width; by default it’s unset and thus disables line-wrapping. If this value is set, you’re entirely at the whimsy of the below formatoptions, which is often filetype sensitive. - formatoptions (or fo): controls whether or not automatic text wrapping is enabled, depending on whether or not the
t flag is set. Toggle the flag on with :set fo+=t, and toggle it off with :set fo-=t. There are also a number of auxiliary format options, but they’re not as important. - wrapmargin (or wm): controls when to wrap based on terminal size; I generally find using this to be a bad idea.
Understanding the interaction between these two options is important. Here is a short table of interactions:
- tw=0 fo=cq wm=0: No automatic wrapping, rewrapping will wrap to 80
- tw=72 fo=cq wm=0: No automatic wrapping, rewrapping will wrap to 72
- tw=0 fo=cqt wm=0: No automatic wrapping, rewrapping will wrap to 72
- tw=0 fo=cqt wm=5: Automatic wrapping at a 5 col right margin
- tw=72 fo=cqt wm=0: Automatic wrapping at col 72
Notice that to get automatic wrapping you need both fo+=t as well as tw or wm to be nonzero. Note also that some filetype will automatically give you fo+=t, while others won’t.
Here are the keystrokes you care about:
- gq: performs a “formatting operation”, which in our universe means “rewrap the text.” This will respect leading indent and symbolic characters, which is usually nice but a little obnoxious if you’re reflowing a bullet point (since the text will suddenly acquire asterisks in front of everything).
- The paragraph motions. The big one is vip (preceding v puts us in visual mode, for selection), which selects an “inner paragraph”; this means that if you’re anywhere inside of a paragraph, you can type vip and have the entire thing instantly selected for you, possibly for you to run gq subsequently. vap is also equivalent, although it selects a whole paragraph and is more appropriate if you want to, say, delete it. The curly braces move you between paragraphs.
The value of format-options will drastically change the way Vim behaves, so I highly recommend keeping it displayed some where you can reference it quickly. I use:
set statusline=...[%{&fo}]...
You probably have a statusline of your own; just add that small snippet minus the ellipses in somewhere convenient. For further good measure, I explicitly say set fo-=t in my vimrc, to prevent myself from being surprised (since I do primarily coding in vim).
One more neat trick:
augroup vimrc_autocmds
autocmd BufEnter * highlight OverLength ctermbg=darkgrey guibg=#592929
autocmd BufEnter * match OverLength /\%74v.*/
augroup END
This will highlight all characters past 74 columns (tweak that number as desired) in dark grey (tweak that color as desired), and is a nice visual cue when auto linewrapping isn’t turned on when you should think about breaking things.
March 3, 2010unattended-upgrades is a nifty little package that will go ahead and automatically install updates for you as they become enabled. No serious system administrator should use this (you are testing updates before pushing them to the servers, right?) but for many personal uses automatic updates are really what you want; if you run sudo aptitude full-upgrade and don’t read the changelog, you might as well turn on unattended upgrades. You can do this by adding the line APT::Periodic::Unattended-Upgrade "1" to /etc/apt/apt.conf.d/10periodic (thanks Ken!)
Of course, the default configuration they give you in /etc/apt/apt.conf.d/50unattended-upgrades only pulls updates from their security repository, and they only give you a commented out line for normal updates. People have asked, “well, how do I pull automatic updates from other repositories?” Maybe you have installed Chromium dailies; seeing the “you have updates” icon every day can be kind of tiresome.
Well, here’s how you do it:
- Find out what URL the PPA you’re interested in points to. You can dig this up by looking at
/etc/apt/sources.list or /etc/apt/sources.list.d/ (the former is if you manually added a PPA at some point; the latter is likely if you used add-apt-repository). - Navigate to that URL in your browser. Navigate to
dists, and then navigate to the name of the distribution you’re running (for me, it was karmic). Finally, click Release. (For those who want to just enter the whole URL, it’s http://example.com/apt/dists/karmic/Release). - You will see a number of fields
Fieldname: Value. Find the field Origin and the field Suite. The two values are the ones to put in Allowed-Origins.
For example, the Ksplice repository has the following Release file:
Origin: Ksplice
Label: Ksplice
Suite: karmic
Codename: karmic
Version: 9.10
Date: Sun, 07 Feb 2010 20:51:12 +0000
Architectures: amd64 i386
Components: ksplice
Description: Ksplice packages for Ubuntu 9.10 karmic
This translates into the following configuration:
Unattended-Upgrade::Allowed-Origins {
"Ksplice karmic";
};
And that’s it! Go forth and make your systems more secure through more timely updates.
Bonus tip. You can turn on unattended kernel updates via Ksplice by editing /etc/uptrack/uptrack.conf and setting autoinstall = yes.
