This post is a long form essay version of a talk about PyTorch internals, that I gave at the PyTorch NYC meetup on May 14, 2019.
Hi everyone! Today I want to talk about the internals of PyTorch.
This talk is for those of you who have used PyTorch, and thought to yourself, “It would be great if I could contribute to PyTorch,” but were scared by PyTorch’s behemoth of a C++ codebase. I’m not going to lie: the PyTorch codebase can be a bit overwhelming at times. The purpose of this talk is to put a map in your hands: to tell you about the basic conceptual structure of a “tensor library that supports automatic differentiation”, and give you some tools and tricks for finding your way around the codebase. I’m going to assume that you’ve written some PyTorch before, but haven’t necessarily delved deeper into how a machine learning library is written.
Read more...
Some notes collected from a close read of Conal Elliot’s Compiling to Categories and The Simple Essence of Automatic Differentiation.
A colleague of mine was trying to define a “tree structure” of tensors, with the hope of thereby generalizing the concept to also work with tensors that have “ragged dimensions.” Let’s take a look:
Suppose we have a (2, 3) matrix:
tensor([[1, 2, 3],
[4, 5, 6]])
One way to think about this is that we have a “tree” of some sort, where the root of the tree branches to two subnodes, and then each subnode branches to three nodes:
Read more...
Long time readers of mine may be aware that I used a ThinkPad X61T for the past decade. After the hinge on my second instance of the machine, I decided it was finally time to get a new laptop. And I had one particular model on my eye, after Simon Peyton Jones showed me his new laptop at the last Haskell Implementor’s Workshop: the Microsoft Surface Book 2. It fits my primary requirement for a laptop: it’s a convertible laptop into tablet mode with a digitizer pen. The pen is not Wacom branded but it has an eraser end and can magnetically attach to the laptop (no enclosure for the pen, but I think that for modern hardware that constraint is unsatisfiable.) Furthermore, there is a Linux enthusiast community around the device, which made me feel that it would be more likely I could get Linux to work. So a few weeks ago, I took the plunge, and laid down three grand for my own copy. It has worked out well, but in the classic Linux style, not without a little bit of elbow grease.
Read more...