ezyang’s blog

the arc of software bends towards understanding

Some thoughts about literature review

While working on my senior thesis, I had to write a prior work section, which ended up being a minisurvey for the particular subfield my topic was in. In the process, a little bird told me some things...

  • If you can, ask someone who might know a little bit about the subject to give you the rundown: there's a lot of knowledge in people's heads which never got written down. But also be aware that they will probably have their blind spots.
  • It is better to do the literature review later rather than earlier, after you have started digging into the topic. I have been told if you read the literature too early, you will get spoiled and stop thinking novel thoughts. But I also think there is also a little bit of "you'll understand the literature better" if you've already thought about the topic on your own. Plus, it's easy to think that everything has been done before: it's simply not true! (But if you think this, you will get needlessly discouraged.)
  • Don't indiscriminately add papers to your database. You should have something you want to do with it: is it an important paper that you have to cite because everyone knows about it? Is it directly addressing the issue you're dealing with? Does it happen to be particularly well written? Is it something that you could see yourself reading more carefully later? Don't be afraid to toss the paper out; if it actually was important, you'll run into it again later.
  • Every researcher is a historian. When you look at a paper, you're not just looking at what is written inside it, but its social context. There's a reason why "prior work" is so important to academics. If you don't understand a paper's context, it's unlikely you'll understand the paper.
  • Researchers don't necessarily talk to each other. Pay attention to who they cite; it says a little bit about what community they're in.
  • Researchers are happy to send you copies of papers they have written (so fear not the paywall that your university hasn't subscribed to). They may even volunteer extra information which may come in handy.
  • Be methodical. You're doing a search, and this means carefully noting down which papers you skimmed, and what you got out of them, and keeping track of other veins of research that you need to follow up on. It's like chasing a rabbit down a hole, but if you have some clearly defined search criteria, eventually you'll bottom out. You can prune the uninteresting papers later; the point here is to avoid duplicating work.
  • Read papers critically. Not everything that is published is good; that's the point of research!

What are your favorite maxims to keep in mind while you're surveying the literature?

5 Responses to “Some thoughts about literature review”

  1. Anonymous says:

    How do you go about the second-to-last bullet, being methodical? I haven’t found a system that works well for me.

  2. Hmm, I don’t think I have a particularly well defined system. Some things that I found work OK:

    – Download all the papers, don’t just open them in the browser. The idea is that they all get shoved in a folder, and when you decide you’re done doing discovery (for now), you can then focus on the papers in the folder. There’s no need to add any metadata, because you’re going to go over all of them again; it’s just important to have them around and all in one place (and not sprawled out over lots of browser tabs.)

    – As *soon* as you decide that a paper is important enough to get cited, you need to add it to whatever paper tracking system you use and make sure the metadata is correct. (I use Mendeley for this, but I’m sure there’s other ways.) Putting papers here is nontrivial effort, so I guess I’m a little more choosy about this step. (I’ve also had lots of bad experiences where I just dumped all the papers in my repo, and then couldn’t remember what all of them meant and just let the stack of unsorted stuff get bigger and bigger.)

    – Keep a stack of deferred operations. Physical notepad works decently well. “Paper to read” is probably too fine grained, but “name of system”, etc is probably the right size. This might work less well in theoryland.

    I don’t have any idea how far up they scale; my survey wasn’t that big. :^)

  3. gasche says:

    > Researchers are happy to send you copies of papers they have written (so fear not the paywall that your university hasn’t subscribed to).

    You should not ask them to send *you* the papers in question, but to *put them on their webpage*. If their paper is only reachable behind a paywall, they’re doing a bad job at disseminating their research. Which absurd copyright policy they may or may not have forcefully agreed to is irrelevant.

  4. Jan Stolarek says:

    I wish I read those advice a couple of years ago when I started my PhD. Now, after over 3 years, I’ve reached similar conclusions. I think it’s important to do literature review at all stages. If you don’t do it early you might end up repeating others work.

  5. gasche: Yes, I’m a bit of a horrible person. ;-)

    There are a lot of reasons why the author may not have published the paper on their website (e.g. copyright issues), in general putting the paper on the website is more work than just attaching it to an email, and all-in-all asking for a copy seems less imposing than demanding that they make the paper free. So while the optimal social outcome is the paper goes online, the individual incentives seem to stack up otherwise.

Leave a Comment