Open Source Emacs Configuration Improvements

In retrospect I'm not totally sure why I released my emacs configuration to the world. I find tweaking Emacs Lisp to be soothing, and in 2020 these kinds of projects are particularly welcome. I've always thought about making it public: I feel like I get a lot out of Emacs, and I'm super aware that it's very hard for people who haven't been using Emacs forever to get a comparable experience. [1]

I also really had no idea of what to expect, and while it's still really recent, I've noticed a few things which are worth remarking:

  • Making your code usable for other people really does make it easy for people to find bugs. While it's likely that there are bugs that people never noticed, I found a few things very quickly:

    • Someone reported higher than expected CPU use, and I discovered that there were a number of functions that ran regularly in timers, and I was able to quickly tune some knobs in order to reduce average CPU use by a lot. This is likely to be great both for the user in question, but also because it'll help battery life.
    • The config includes a git submodule (!) with the contents of all third-party packages, mostly to reduce friction for people getting started. Downloading all of the packages fresh from the archive would take a few minutes, and the git clone is just faster. I realized, when someone ran into some problems when running with emacs 28 (e.g. the development/mainline build,) that the byte-compilation formats were different, which made the emacs27 files not work on emacs28. I pushed a second branch.

    More than anything the experience of getting bug reports and feedback has been great. It both makes it possible to focus time because the impact of the work is really clear, and it also makes it clear to me that I've accumulated some actually decent Emacs Lisp skills, without really noticing it. [2]

  • I was inspired to make a few structural improvements.

    • For a long time, including after the initial release, I had a "settings" file, and a "local functions" file that held code that I'd written or coppied from one place or another, and I finally divided them all into packages named tychoish-<thing>.el which allowed me to put all or most of the configuration into use-package forms, which is more consistent and also helps startup time a bit, and makes the directory structure a bit easier.
    • I also cleaned up a bunch of local snippets that I'd been carrying around, which wasn't hurting anything but is a bit more clear in the present form.
  • I believe that I've hit the limit, with regards to startup speed. I'd really like to get a GUI emacs instance to start (with no buffers) in less than a second, but it doesn't seem super plausible. I got really close. At this point there are two factors that constrain:

    • Raw CPU speed. I have two computers, and the machine with the newer CPU is consistently 25% faster than the slow computer.
    • While the default configuration doesn't do this, my personal configuration sets a font (this is reasonable,) but seems that the time to do this is sometimes observable, and proportional to the number of fonts you have installed on the system. [3]
    • Dependencies during the early load. I was able to save about 10% time by moving a function between package to reduce the packages that startup code depended upon. There's just a limit to how much you can clean up here.

    Having said that, these things can drift pretty easily. I've added some helper macros with-timer and with-slow-op-timer that I can use to report the timing of operations during startup to make sure that things don't slow down.

    Interestingly, I've experimented with byte-compiling my local configuration and I haven't really noticed much of a speedup at this scale, so for ease I've been leaving my own lisp directory unbytecompiled.

  • With everything in order, there's not much to edit! I guess I'll have other things to work on, but I have made a few improvements, generally:

    • Using the alert package for desktop notification, which allowed me to delete a legacy package I've been using. Deleting code is awesome.
    • I finally figured out how to really take advantage of projectile, which is now configured correctly, and has been a lot of help in my day-to-day work.
    • I've started using ERC more, and only really using my irssi (in screen) session as a fallback. My IRC/IM setup is a bit beyond the scope of this post but ERC has been a bit fussy to use on machines with intermittent connections, but I think I've been able to tweak that pretty well and have an experience that's quite good.

It's been interesting! And I'm looking forward to continuing to do this!

[1]Sure, other editors also have long setup curves, but Emacs is particularly gnarly in this regard, and I think adoption by new programmers is definitely constrained by this fact.
[2]I never really thought of myself as someone who wrote Emacs Lisp: I've never really written a piece of software in Emacs, it's always been a function here or there, or modifying some snippet from somewhere. I don't know if I have a project or a goal that would involve writing more emacs software, but it's nice to recognize that I've accidentally acquired a skill.
[3]On Windows and macOS systems this may not matter, but you may have more fonts installed than you need. I certianly did. Be aware that webbrowsers often downlaod their own fonts separately from system fonts, so having fonts installed is really relevant to your GTK/QT/UI use and not actually to the place where you're likely doing most of your font interaction (e.g. the browser.)

Knitting Pictures

I've never been really good at the blogging+picture game, and while maybe once upon a time it was technical limitation--taking photos and getting them online was complicated--anymore it's probably not. To this end, I've started a knitting specific Instagram account as a kind of photoblog for knitting things. It's @gestaltknitting, if you're interested.


While I took this picture a while ago, I must confess that my knitting basically looks the same now.

The same, not because I've made no progress, but because sleeves take a while and it's just plain knitting, so unless you have a very discerning eye, you might miss the details.

Indeed, I really want my next project to also have a lot of plain knitting with black yarn: I expect the photographs will be captivating. Perhaps it will be enjoyable for people to be able to spot the different patterns of embedded cat hair in the sweaters.


I get that knitting is visual for a lot of people, and I do like a smart looking sweater as much as the next guy, but I've always felt somewhat resistant to this view: knitting is about the process and the act more than it is about the product, and so the things that are most exciting aren't the visuals.

While it's gotten much easier to take high quality pictures, my intention for this book that I've been writing is that it mostly would not be a book with a lot of picture, though we'll see: If anything, I suspect that diagrams and cartoons may be more effective for this kind of application.

Having said that, it's nice to see what other people are knitting, and I like the way that the ephemeral nature of instagram stories make it less daunting to post in-progress updates on projects. So I've definitely been enjoying that.

We'll see!

Learning Common Lisp Again

In a recent post I spoke about abandoning a previous project that had gone off the rails, and I've been doing more work in Common Lisp, and I wanted to report a bit more, with some recent developments. There's a lot of writing about learning to program for the first time, and a fair amount of writing about lisp itself, neither are particularly relevant to me, and I suspect there may be others who might find themselves in a similar position in the future.

My Starting Point

I already know how to program, and have a decent understanding of how to build and connect software components. I've been writing a lot of Go (Lang) for the last 4 years, and wrote rather a lot of Python before that. I'm an emacs user, and I use a Common Lisp window manager, so I've always found myself writing little bits of lisp here and there, but it never quite felt like I could do anything of consequence in Lisp, despite thinking that Lisp is really cool and that I wanted to write more.

My goals and rational are reasonably simple:

  • I'm always building little tools to support the way that I use computers, nothing is particularly complex, but it'd enjoy being able to do this in CL rather than in other languages, mostly because I think it'd be nice to not do that in the same languages that I work in professionally. [1]
  • Common Lisp is really cool, and I think it'd be good if it were more widely used, and I think by writing more of it and writing posts like this is probably the best way to make that happen.
  • Learning new things is always good, and I think having a personal project to learn something new will be a good way of stretching my self as a developer. Most of my development as a programmer has focused on
  • Common Lisp has a bunch of features that I really like in a programming language: real threads, easy to run/produce static binaries, (almost) reasonable encapsulation/isolation features.

On Learning

Knowing how to program makes learning how to program easier: broadly speaking programming languages are similar to each other, and if you have a good model for the kinds of constructs and abstractions that are common in software, then learning a new language is just about learning the new syntax and learning a bit more about new idioms and figuring out how different language features can make it easier to solve problems that have been difficult in other languages.

In a lot of ways, if you already feel confident and fluent in a programming language, learning a second language, is really about teaching yourself how to learn a new language, which you can then apply to all future languages as needed.

Except realistically, "third languages" aren't super common: it's hard to get to the same level of fluency that you have with earlier languages, and often we learn "third-and-later" languages are learned in the context of some existing code base or project4, so it's hard to generalize our familiarity outside of that context.

It's also the case that it's often pretty easy to learn a language enough to be able to perform common or familiar tasks, but fluency is hard, particularly in different idioms. Using CL as an excuse to do kinds of programming that I have more limited experience with: web programming, GUI programming, using different kinds of databases.

My usual method for learning a new programming language is to write a program of moderate complexity and size but in a problem space that I know pretty well. This makes it possible to gain familiarity, and map concepts that I understand to new concepts, while working on a well understood project. In short, I'm left to focus exclusively on "how do I do this?" type-problems and not "is this possible," or "what should I do?" type-problems.

Conclusion

The more I think about it, the more I realize that when we talk about "knowing a programming language," inevitably linked to a specific kind of programming: the kind of Lisp that I've been writing has skewed toward the object oriented end of the lisp spectrum with less functional bits than perhaps average. I'm also still a bit green when it comes to macros.

There are kinds of programs that I don't really have much experience writing:

  • GUI things,
  • the front-half of the web stack, [2]
  • processing/working with ASTs, (lint tools, etc.)
  • lower-level kind of runtime implementation.

There's lots of new things to learn, and new areas to explore!

Notes

[1]There are a few reasons for this. Mostly, I think in a lot of cases, it's right to choose programming languages that are well known (Python, Java+JVM friends, and JavaScript), easy to learn (Go), and fit in with existing ecosystems (which vary a bit by domain,) so while it might the be right choice it's a bit limiting. It's also the case that putting some boundaries/context switching between personal projects and work projects could be helpful in improving quality of life.
[2]Because it's 2020, I've done a lot of work on "web apps," but most of my work has been focused on areas of applications including including data layer, application architecture, and core business logic, and reliability/observability areas, and less with anything material to rendering web-pages. Most projects have a lot of work to be done, and I have no real regrets, but it does mean there's plenty to learn. I wrote an earlier post about the problems of the concept of "full-stack engineering" which feels relevant.

Pattern Fragment 0

I was doing some knitting pattern math, [1] and I thought I'd share it without a lot of context:

Cast on 228 stitches using the "German Twisted" method, [2], placing a marker half way, after 114 stitches. Knit 2 inches of knit 1 purl 1 ribbing.

After two inches, switch to stocking stitch: knit 21 stitches, increase 1 stitch, place a marker, knit 72 stitches, place a marker, increase 1 stitch, knit 21 more stitches. You should have arrived at the "half way" marker from before.

Over the next half (115 stitches), space out 14 increases. This doesn't divide evenly, so try: knit 5, increase 1 stitch and then knit 8 stitches, increase 1 stitch 13 times, or in short hand: K5 M1, * K8 M1, repeat from * 13 times, K5).

The "first half" is the back of the sweater and the "second" half is the front. Increase one stitch before and after the markers on the back of the sweater 7 times, every 1.5 or 2 inches (somewhere between 10 or 20 rows,) depending on how you'd like the taper.

Meanwhile [3] insert 3 sets of short rows across the back of the sweater, which should get wider. For the first short row stop 3 inches from the edges, for the second 2 inches, and for the last 1 inch. I'd put an inch or two between each short row, maybe half way between the first three increases.

Notes

[1]I've not, to be clear, actually knit this yet, though I plan to soon.
[2]As in this video, though there are many videos that may be more clear for you. I'm pretty sure that learned this method from Meg Swansen and/or Amy Detjin.
[3]I have to say, that the "meanwhile" part of knitting patterns is always my favorite.

Reknitting Projects

I'm presently in the middle of knitting a sweater that I knit and designed years and years ago, with only minor modifications, and I have a number of projects that I'm thinking about that involve "reknitting" past projects. While I don't think that I've peaked, or am out of ideas for knitting, it's very clear to me that novelty isn't exactly my guiding principle as a knitter: I enjoy the process and the act above all else, and the pleasure of wearing handknits is (for me) mostly about custom fit and less about novelty or fashion, exactly.

The chance to re-knit things, removes a lot of the questions of a design from the process and not only fix mistakes, but also polish and iterate on a garment with less guess work. It's also the case that these projects often feel like returning to an old friend, which is incredibly comforting. Some of these projects, on my backlog include:

  • This basic two-color sweaters (colorblock, I suppose,) that I'm presently knitting and have/will knit again where the lower part of the body is in black--or similar very dark--except for the top 3-4 inches of the body in a contrasting color, matched by the sleeves and the collar, which I try and push into the black section.
  • Alice Starmore's Faroe Sweater, from Fishermen's Sweaters, but scaled to actually fit and maybe with a more fitted shoulder. I've also, apparently knit a very heavy weight version of the Norway sweater that I never wore, and they're such great classic designs that are very fun to knit that knitting them again to modernize them sounds like a fun project.
  • A round pi shawl in a dark color, with no lace work (including using raised bar increases rather than yarn overs), and a contrasing set of stripes along the outer edge. There's this stripe pattern that I think of as "Calvin Klein" stripes, but I don't kno what the origin of that association is, the basic plan is three stripes, two wide stripes in the contrasting color, and a thin stripe of the original color in between, with the wide stripes being 3 times the width of the interior stripe.
  • I've knit two sweaters from Joyce Willams' Latvian Dreams book, the sweater on the cover and one that I knit from several charts, using yarn that ended up pilling a lot. They were delightful to knit: the patterns were originally weaving charts rather than knitting patterns, and thus had a 4-way radial summary symmetry that was just fun to knit. I'd like to try some of these again with better yarn and perhaps use this as a space to explore color work again, but in ways that might be more subtle and also well suited to cardigans and the like.
  • I've knit a handful of sweaters with all-over mitten or stocking patterns from various extant knitting traditions, mostly Scandinavian and Turkish, and I think it would be fun to revisit these patterns.

For and Against Garter Stitch

I never used to like garter stitch [1] very much, and hadn't really knit things with a lot of garter stitch. Sure, a scarf here or there in the beginning, and I think I used it for the hem of an early sweater that didn't turn out particularly well. There are so many clever patterns that use a lot of garter stitch, and I'd never really felt it. While I don't know that I'm rushing to knit or design patterns out of a lot of garter stitch, I've definitely discovered that I've softened on it over my hiatus.

My earlier discontent with garter stitch was the combination of:

  • garter stitch is quite dense, because the fabric pulls in so much vertically, so it takes a lot of yarn and a lot of time, and results in a warmer fabric that I often don't like'
  • the vertical pull in of the fabric can get pulled out by blocking or by the weight of the fabric which can be rather uneven.
  • normal tension irregularity is super apparent.
  • I've never much liked the way that knitting things with rows require you to flip the knitting and I don't like the way that this can break up the rhythm of the knitting.
  • the strong horizontal line of the garter ridges always feels awkward to work with.
  • I always struggled to get a selvage edge that I really liked that wasn't totally sloppy.

These, however, are tractable problems I realized, and I've always used a few garter stitches for selvage on the edge of sock heel flaps. The things that I've realized:

  • garter stitch often works best with very fine yarn, which helps ameliorate the additional bulk, and at least for me, helps provide for more even tension.
  • the look of garter stitch sideways is quite compelling, for me, and in most cases it won't stretch out in the same way.
  • a little bit goes a long way, particularly when embedded in another piece of knitting.
  • I've settled down and find that knitting, rather than slipping, the first stitch and giving the yarn a slight tug when knitting the second stitch leads to a pretty clean edge.
  • designing with garter stitch is quite compelling, because the ratio of stitches to rows is basically 2:1, because of the way the ridges pull in, you can sort of approach it as "square," picking up one stitch for every garter ridge lays very flat, so the math is never very complicated.

I'm working on a hat where I knit a ~2 inch wide garter stitch strip to fit around my head and then picked up to knit the crown of the hat along one of the sides of the strip, and along the other to knit a lining. I could have used a provisional cast on, of course, but the strip allowed me to be more confident about sizing, and it ends up being pretty sharp.

I'm not sure I'm going to plan to knit things out of primarily garter stitch, but I've definitely softened rather a lot.

[1]The fabric that results from knitting all stitches on both the front and back of the fabric. The fabric is dense, and it grows slowly, because the "ridges" account for two rows of knitting and it pulls in rather a lot.

Sweater Measurements

Hand knitting provides the opportunity to customize sizing and shaping to fit your body (or that of whomever you're knitting for,) and it's possible to produce garments that really fit, but even though it's possible it's not always easy.

First, measuring a body directly is complicated:

  • posture impacts the measurements, and it's difficult to get measurements of the body in the kinds of shapes and positions that you're likely to hold while wearing the garment.
  • ease, or the difference between the actual measurement of your body and the actual measurement of the garment, is both subjective and a matter of preference.

For this reason, I normally recommend measuring another sweater that has a fit that you enjoy as a starting point, but there are challenges:

  • measurements for different styles of sweaters can have different internal proportions: the length of the sleeve depends on the width of the shoulders, and the depth of the armhole
  • most machine produced garments and conventional knitting patterns are based on typical measurements and proportions which are good as starting points but typically leave something to be desired.

While people's measurements are broadly similar, and proportional, they're not the same, so if you have slightly longer arms or shoulders that are a bit more broad or angular, the "average" might be off by an inch or two, which might be enough to care about.

I'd still recommend starting from a garment that you know fits well, and record the garment's measurements as clearly as possible, but also note modifications separately. The basic idea is lay the garment out as flat as possible and measure the garment which is less likely to move than a person. There are three or four measurements that are really critical:

  • width of body at across the chest below the arms.
  • width of the body at the bottom hem/edge.
  • distance from the middle of the back of the neck to the cuff.
  • length of the sweater from the top of the shoulder to the bottom hem.

Sleeve length is pretty stable when measured from the bottom of the sleeve (where it joins the body at the underarm) to the cuff, as this avoids the impact of shoulder shape on the sleeve. Measuring arm length from a common point, the middle back of the neck, to the cuff is also a stable way to take this measurement. You may also require additional measurement's if you want the body of the garment to have contores.

While it's true that you can deduce other measurements from the four basic measurements, there are other fit considerations that are worth noting: width of the sleeve at/above the cuff and at the shoulder; depth, height, and aperture of the collar; as well as "true" shoulder width. May of these details I've figured out empirically and iteratively for myself: it's sometimes difficult to get these measurements correctly from a model garment.

Better Company

I've been using company mode, which is a great completion framework for years now, and in genreal, it's phenomenal. For a while, however, I've had the feeling that I'm not getting completion options at exactly the right frequency that I'd like. And a completion framework that's a bit sluggish or that won't offer completions that you'd expect is a drag. I dug in a bit, and got a much better, because of some poor configuration choices I'd made, and I thought I write up my configuration.

Backend Configuration

Company allows for configurable backends, which are just functions that provide completions, many of which are provided in the main company package, but also provided by many third (fourth?) party packages. These backends, then, are in a list which is stored in the company-backends, that company uses to try and find completions. When you get to a moment when you might want to complete things, emacs+company iterate through this list and build a list of expansions. This is pretty straight forward, at least in principle.

Now company is pretty good at making these backends fast, or trying, particularly when the backend might be irrelevant to whatever you're currently editing--except in some special cases--but it means that the order of things in the list matters sometimes. The convention for configuring company backends is to load the module that provides the backend and then push the new backend onto the list. This mostly works fine, but there are some backends that either aren't very fast or have the effect of blocking backends that come later (because they're theoretically applicable to all modes.) These backends to be careful of are: company-yasnippet, company-ispell, and company-dabbrev.

I've never really gotten company-ispell to work (you have to configure a wordlist,) and I've never been a dabbrev user, but I've definitely made the mistake to putting the snippet expansion near the front of the list rather than the end. I've been tweaking things recently, and have settled on the following value for company-backends:

(setq company-backends '(company-capf
                         company-keywords
                         company-semantic
                         company-files
                         company-etags
                         company-elisp
                         company-clang
                         company-irony-c-headers
                         company-irony
                         company-jedi
                         company-cmake
                         company-ispell
                         company-yasnippet))

The main caveat is that everything has to be loaded or have autoloads registered appropriately, particularly for things like jedi (python,) clang, and irony. The "capf" backend is the integration with emacs' default completion-at-point facility, and is the main mechanism by which lap-mode interacts with company, so it's good to keep that at the top.

Make it Fast

I think there's some fear that a completion framework like company could impact the perceived responsiveness of emacs as a whole, and as a result there are a couple of knobs for how to tweak things. Having said that, I've always run things more aggressively, because I like seeing possible completions fast, and I've never seen any real impact on apparent performance or battery utilization. use these settings:

(setq company-tooltip-limit 20)
(setq company-show-numbers t)
(setq company-idle-delay 0)
(setq company-echo-delay 0)

Configure Prompts

To be honest, I mostly like the default popup, but it's nice to be able to look at more completions and spill over to helm when needed. It's a sometimes thing, but it's quite nice:

(use-package helm-company
   :ensure t
   :after (helm company)
   :bind (("C-c C-;" . helm-company))
   :commands (helm-company)
   :init
   (define-key company-mode-map (kbd "C-;") 'helm-company)
   (define-key company-active-map (kbd "C-;") 'helm-company))

Full Configuration

Some of the following is duplicated above, but here's the full configuration that I run with:

(use-package company
  :ensure t
  :delight
  :bind (("C-c ." . company-complete)
         ("C-c C-." . company-complete)
         ("C-c s s" . company-yasnippet)
         :map company-active-map
         ("C-n" . company-select-next)
         ("C-p" . company-select-previous)
         ("C-d" . company-show-doc-buffer)
         ("M-." . company-show-location))
  :init
  (add-hook 'c-mode-common-hook 'company-mode)
  (add-hook 'sgml-mode-hook 'company-mode)
  (add-hook 'emacs-lisp-mode-hook 'company-mode)
  (add-hook 'text-mode-hook 'company-mode)
  (add-hook 'lisp-mode-hook 'company-mode)
  :config
  (eval-after-load 'c-mode
    '(define-key c-mode-map (kbd "[tab]") 'company-complete))

  (setq company-tooltip-limit 20)
  (setq company-show-numbers t)
  (setq company-dabbrev-downcase nil)
  (setq company-idle-delay 0)
  (setq company-echo-delay 0)
  (setq company-ispell-dictionary (f-join tychoish-config-path "aspell-pws"))

  (setq company-backends '(company-capf
                           company-keywords
                           company-semantic
                           company-files
                           company-etags
                           company-elisp
                           company-clang
                           company-irony-c-headers
                           company-irony
                           company-jedi
                           company-cmake
                           company-ispell
                           company-yasnippet))

  (global-company-mode))

(use-package company-quickhelp
  :after company
  :config
  (setq company-quickhelp-idle-delay 0.1)
  (company-quickhelp-mode 1))

(use-package company-irony
  :ensure t
  :after (company irony)
  :commands (company-irony)
  :config
  (add-hook 'irony-mode-hook 'company-irony-setup-begin-commands))

(use-package company-irony-c-headers
  :ensure t
  :commands (company-irony-c-headers)
  :after company-irony)

(use-package company-jedi
  :ensure t
  :commands (company-jedi)
  :after (company python-mode))

(use-package company-statistics
  :ensure t
  :after company
  :config
  (company-statistics-mode))