Pragmatic Emacs: Google search from inside emacs

Artur Malabarba, of endless parentheses, has a nice package called google this which provides a set of functions for querying google from emacs.

Here is my code to install google-this

;; google-this
(use-package google-this
  (google-this-mode 1))

The package provides a set of functions under the prefix C-c /. The simplest is C-c / RET which prompts you for a search in the minibuffer, with a default search string based on the text around the point. This sounds trivial, but I find myself using it all the time as it is more efficient than switching to my browser, moving to the search box and trying the search string.

The full list of commands is:

C-c / SPC google-this-region
C-c / a google-this-ray
C-c / c google-this-translate-query-or-region
C-c / e google-this-error
C-c / f google-this-forecast
C-c / g google-this-lucky-search
C-c / i google-this-lucky-and-insert-url
C-c / l google-this-line
C-c / m google-maps
C-c / n google-this-noconfirm
C-c / r google-this-cpp-reference
C-c / s google-this-symbol
C-c / t google-this
C-c / w google-this-word
C-c / <return> google-this-search
-1:-- Google search from inside emacs (Post Ben Maughan)--L0--C0--July 29, 2016 01:28 PM

(or emacs: Swipe all the files!

If you've ever tried the swiper-all from swiper, forget everything about it. The command was super-awkward, since it had to parse all your open files before giving you a chance enter anything, resulting in dozens of seconds before the prompt.

Recently, I've had some time to examine and improve it and the result looks very promising. The new command is now async, which means there's no delay before the prompt comes up. Here's a result I got with no delay while having around 50 buffers open:


The shortcut I'm using:

(global-set-key (kbd "C-c u") 'swiper-all)

For efficiency's sake a small trade off had to be made: the line numbers are no longer displayed. This actually results in an advantage that you can select different candidates on the same line.

There are still a few things I plan to try for the new command, like adding file-less buffers, caching for incremental regexes and maybe even newlines in wild cards, but even now it seems very usable. So give it a try, enjoy and happy hacking!

-1:-- Swipe all the files! (Post)--L0--C0--July 28, 2016 10:00 PM

Irreal: New Zamansky Emacs Video Up

Mike Zamansky has posted the latest Emacs video in his series. This time it's about flycheck. As usual, the video is from the point of view of how Zamansky himself uses Emacs.

The demonstration is mostly about using flycheck with Python because that's the language that Zamansky uses most of the time but he also shows an example of using it with C. He points to the flycheck home page and the list of supported languages, which includes Elisp, of course.

I often see queries on how to set up a Python environment in Emacs. Zamansky discusses that a bit too but doesn't cover Elpy because it's more powerful than he needs. He does talk about Pylint, Flake8, and Jedi, though. If you're looking for a Python environment, check out this video. Those of you who haven't tried flycheck yet should check it out too.

-1:-- New Zamansky Emacs Video Up (Post jcs)--L0--C0--July 28, 2016 08:13 PM

Irreal: The Emacs/Vi Holy War

I saw this tweet and it got me thinking.

The tweet is, of course, snark but it raises an interesting question. Does anyone still care about the holiest of holy wars? This tweet

suggests that some of us do but my sense is that Emacsers and Vimers are pretty much united against all the newcomers as exemplified by this tweet

I always get in trouble when I bring this up but I think it's true that serious developers overwhelmingly prefer either Emacs or Vim. Of course there are exceptions. There are, I'm sure, thousands of excellent developers that use something else but mostly the great developers use Emacs or Vim.

The choice between the two depends on the developer's outlook. If you want the fastest, most composable editor and are focused on simply editing text, you will probably prefer Vim. If, on the other hand, you want an environment that subsumes editing among other things, you will probably prefer Emacs.

My point, though, is that Emacsers and Vimers have pretty much moved from fighting each other to shaking their heads in disbelief about those engineers who are using one of those other editors. So perhaps the holy war isn't over, it's just move to another domain.

-1:-- The Emacs/Vi Holy War (Post jcs)--L0--C0--July 28, 2016 01:36 AM

John Stevenson: Spacemacs - Adding Custom Snippets to Yasnippet

Using yasnippet saves time by avoiding the need to write boilerplate code and minimising other commonly typed content. YASnippet contains mode-specific snippets that expand to anything from a simple text replacement to a code block structure that allows you to skip through parameters and other sections of the code block. See YASnippet in action in this Emacs Yasnippet video.

To use a specific snippet simply type the alias and press M-/. For example, in html-mode typing div and pressing M-/ expands to <div id="▮" class="▯">▯</div> and places the cursor so you can type in the id name, then TAB to the class name, finally TAB to the contents of the div.

You can also combine yasnippets with autocompletion select snippets from the autocompletion menu.

Spacemacs has lots of snippets for most of the languages and modes it supports. However, YASnippets also uses a simple template system in plain text, so its pretty easy to learn. Lets look at how to add your own snippets with Spacemacs.

In regular Emacs, yasnippets expand funciton is usually bound to TAB, but that key is used already in Spacemacs so M-/ is used instead.
If you just want text replacement you can also use Emacs Abbrev mode.

Adding your private snippets to Spacemacs

The easiest place to add your own snippet definitions is in the ~/.emacs.d/private/snippets directory. Under this directory structure you should create a folder named after the relevant mode for your snippets, eg markdown-mode. Inside this mode folder, create files whos names are based on the snippet alias you wish.

So for a work in progress snipped called wip in markdown mode I created ~/.emacs.d/private/snippets/markdown-mode/wip file.

You need to load this new snippet into Spacemacs by either restarting or using the command M-x yas-load-snippet-buffer command in the buffer of the new snippet you have just written. Ths snippet with then work within any markdown mode buffer.

Managing your snippets

Although the private snippets directory is easy to use, it is not under version control. So although its not over-riddend by Spacemacs it also means your private snippets are not backed up anywhere.

If you use the ~/.spacemacs.d/snippets/modename-mode/ directory structure for your snippets then you can version them with Git or similar versioning tools.

How to write a snippet

Typically each snippet template is contained in its own file, named after the alias of the snippet. So a snippet called wip will be in a filename wip, in a directory named after the relevant Emacs mode.

The basic structure of a snippet template is:

#key : the name of the snippet you type
#name : A description of the snippet (this shows in autocompletion menu too)
#contributor: John Stevenson <>
# --
Add the content you want to replace the snippet name with when it expands

The content can be anything, simple text or more usefully a code strucuture with placeholders for tab stops. You can even include Emacs lisp (elisp) code in there too.

Example: Simple text replacement

I use markdown mode for writing a lot of content, especially for technical workshops. As I am developing these workshops its useful to highlight which sections are still work in progress. Rather than type the common message I use, I’ve created a simple snippet called wip.

#key : wip
#name : WorkInProgress
#contributor: John Stevenson <>
# --
> **Fixme** work in progress

When you expand this snippet with M-/ then the snippet name is replaced by the content.

Example: Using tab stops

Lets look at an existing snippet called form in the html-mode. This expands into a html form, but also helps you jump from method, id, action and content.

#contributor : Jimmy Wu <>
#name :<form method="..." id="..." action="..."></form>
# --
<form method="$1" id="$2" action="$3">

This snippet is the same as the simpler example, except we have added tab stops using the $ sign and a number. When you expand this snippet, the snippet name is replaced by the content as usual but the cursor is placed at the first tab stop $1. Each time you press TAB you move to the next tab stop.

$0 is our exit point from the snippet, so pressing TAB reverts to the usual behaviour outside of YASnippet.

Testing your snippets

Once you have written your snippet, you can quickly test it using M-x yas-tryout-snippet. This opens a new empty buffer in the appropriate major mode and inserts the snippet so you can then test it with M-/.

If you just want to try the snippet in an existing buffer, then use M-x yas-load-snippet-buffer to load this new snippet into the correct mode. M-x yas-load-snippet-buffer does exactly the same except it kills the snippet buffer (prompting to save first if neccessary).

There are no default keybindings for these commands in Spacemacs, so you could create a binding under C-o, for example C-o C-s t to try a snippet and C-o C-s l to load a snippet.

Adding yas-snippets to autocompletion in Spacemacs

By adding the autocompletion layer in Spacemacs the YASnippets can be shown in the autocompletion menu as you type.

By default, snippets are not shown in the auto-completion popup, so set the variable auto-completion-enable-snippets-in-popup to t.

(setq-default dotspacemacs-configuration-layers
&apos((auto-completion :variables
auto-completion-enable-snippets-in-popup t)))


Find out more about YASnippets and autocompletion from the Github repository for Spacemacs autocompletion layer.

For more details and examples on writing your own snipplets, take a look at:

Thank you.

-1:-- Spacemacs - Adding Custom Snippets to Yasnippet (Post)--L0--C0--July 27, 2016 07:59 PM

sachachua: 2016-07-25 Emacs News

Links from, /r/orgmode, Hacker News,, Youtube, the changes to the Emacs NEWS file, and emacs-devel.

Past Emacs News round-ups

The post 2016-07-25 Emacs News appeared first on sacha chua :: living an awesome life.

-1:-- 2016-07-25 Emacs News (Post Sacha Chua)--L0--C0--July 25, 2016 11:57 PM

Marcin Borkowski: Making directories on the fly

Today, a friend asked me how to create a directory – or a hierarchy of them – while finding a file in a nonexistent directory. I found some clues on the ‘net, but the best one used the old advice mechanism. So I decided to do it myself, based on the hints found. It turned out to be simpler than I thought; it’s made even easier by the fact that Elisp’s make-directory function acts basically like mkdir -p when given non-nil second (optional) argument.
-1:-- Making directories on the fly (Post)--L0--C0--July 25, 2016 08:36 PM

Raimon Grau: slack -> jabber -> irc -> erc

Slack can be used though alternative gateways appart from the slack clients themselves.  They offer both jabber and irc gateways that map quite well to slack model.

I had previously used the irc gateway, and erc did the trick with no extra configuration at all. But things changed and now I only have the jabber gateway at my disposal.

The problem is that if you want to use a decent emacs client, you'll have to use bitlbee and end up mapping jabber to irc, so you can use erc/circe/rcirc (which are far better at IRC protocol than jabber.el is at xmpp)

To chat privately other peers it's a matter of /j name-of-the-buddy.  Joining rooms is not as direct, and first you have to create some 'mapping' rooms in your bitblee configuration, before being able to join them.

Whatever, here's how to integrate slack with bitlbee (and with erc)

On bitlbee buffer:
  •  chat add 0
 0 stands for the account number. 'conference' has  to be there. it's a plain string, you don't have to substitute it for anything. replace myserver-foo for your server name. room-name is the name of the chat in slack

  • chat list
This will show you the rooms available

  • /join room-name
Notice no # sign.
  • channel 4 del
4 is the chat you want to remove in case you messed up with something.

And that's it. Now you can slack from erc, and you can use all erc machinery (yes erc-robot, I'm looking at you).

-1:-- slack -&gt; jabber -&gt; irc -&gt; erc (Post Raimon Grau ( 25, 2016 05:13 PM

Grant Rettke: Org-Mode 8.3.5 Is Released

Hear, hear! Org-Mode 8.3.5 Is Released!

-1:-- Org-Mode 8.3.5 Is Released (Post Grant)--L0--C0--July 23, 2016 07:45 PM

Pragmatic Emacs: Use visible bookmarks to quickly jump around a file

The visible bookmarks package lets you bookmark positions in the current buffer and then jump between them. This is really useful if you are working on large files and have a few common locations you want to keep returning to. It is more efficient than jumping around the mark ring or moving through edit points.

Here is my code to install and configure visible bookmarks with the recommended key bindings:

(use-package bm
  :bind (("<C-f2>" . bm-toggle)
         ("<f2>" . bm-next)
         ("<S-f2>" . bm-previous)))

Here is a trivial illustration where I use <C-f2> to place three bookmarks (causing those lines to be highlighted) and then <f2> and <S-f2> to move up and down through them, before using <C-f2> again to toggle the bookmarks off.


The package will also let you cycle through bookmarks in multiple buffers, and move in order of “last in first out”. See the documentation for more information.

-1:-- Use visible bookmarks to quickly jump around a file (Post Ben Maughan)--L0--C0--July 22, 2016 12:59 PM

Phil Hagelberg: in which a spacefaring text editor holds up against an onslaught

I've been meaning to write about my latest project Bussard for a while now. I describe it as a "spaceflight programming adventure", but it might be easier to think of it as "Emacs in space, but with a sci-fi novella in it", written in Lua with the LÖVE engine. There's a lot to tell about the game and how I want it to eventually be a way for newcomers to learn programming, but I want to write here about a particular part of the development I had a lot of fun with.

The game is played by interacting with your ship's onboard computer. Naturally because I wanted to give the player the freedom to customize the interface as much as possible, I've modeled it on Emacs. The game starts with your ship in orbit around a star and hoping to intercept an orbiting space station, but once you poke around a bit you realize that "flight mode" is only one of many modes you can activate in your onboard computer.

Pressing ctrl-o allows you to open any file in the in-game computer, and pressing ctrl-enter opens a Lua repl buffer that uses the same editor infrastructure but lets you write code interactively and explore your ship's computer's API. Communicating with space station and planet port computers is done over an SSH client that also lives inside the editor. But all the various modes of the editor are configured with Lua code that runs in user-space; basically that code defines keyboard controls which simply invoke functions of your ship's computer to edit text[1], open SSH connections, engage the ship's engine, etc. Every action you can take is just a Lua function call.

-- bind is for commands that only call their functions once even when held.
-- it takes the name of a mode, a key combo, and a function to run when pressed.
bind("flight", "escape", ship.ui.pause)

-- you can bind keys to existing functions or inline functions.
bind("flight", "ctrl-return", function()

-- the mouse wheel is handled just like any other key press.
bind("flight", "wheelup", zoom_in)
bind("flight", "wheeldown", zoom_out)

-- regular tab selects next target in order of distance from the star.
bind("flight", "tab", ship.actions.next_target)

This is a fantastically flexible model for rich interaction—it can be completely rewritten on the fly, and it's seamless to experiment with new ideas you think might support a better way of doing things. No recompiling, no restarting, just flow. But another benefit of the editor API is that you can call it against in an automated context, such as a headless run[2] that does fuzz tests.

I had to make a few changes to the API for this to work nicely, but in the end I realized they made the system a lot more consistent anyway. The fuzz testing uncovered a nice set of nasty edge-case editor bugs that were not too difficult to fix, but would have taken a lot of time to uncover with manual testing.

local function fuzz(n)
   -- need to display the seed so we can replay problematic sequences
   local seed = tonumber(os.getenv("BUSSARD_FUZZ_SEED") or os.time())
   print("seeding with", seed)

   for i=1,n do
      local mode = editor.mode()
      -- smush together all the different sub-maps (ctrl, alt, ctrl-alt)
      local commands = lume.concat(vals(, vals(mode.ctrl),
                                   vals(mode.alt), vals(mode["ctrl-alt"]))
      local command = lume.randomchoice(commands)

      print("run " .. binding_for(mode, command) .. " in mode " ..
      try(lume.fn(editor.wrap, command))

      -- sometimes we should try inserting some text too
      if(love.math.random(5) == 1) then
         try(lume.fn(editor.handle_textinput, random_text()))

Of course, this is pretty limited in the kinds of bugs it can catch—only problems that result in crashes or hangs can be identified by the fuzz tests. But it gives me confidence when I make further changes if I can throw 32768 cycles of random commands at it without seeing it break a sweat. And it's even better when every incoming patch automatically has the testing applied against it using GitLab's CI.

Stay tuned for a second beta of Bussard to be released very soon! There is still a lot more I want to do with the story line and missions, but the engine is getting more and more polished with each milestone. Feedback is very welcome, as are contributions.

Update: I found that the fuzzer above has a critical flaw: it does not inspect the current mode's parent mode to look for commands there. (For instance, the console's parent mode is edit, and the ssh mode's parent is the console.) Fixing this immediately uncovered four new bugs.

[1] Yes, I know I just set myself up for the old "Bussard is a great OS, it just lacks a decent text editor" joke. Honestly I am just waiting for someone to come along and implement a vim mode in-game; if any player thinks they can do better than the built-in editor they are welcome to try!

[2] It's a bit tricky to get LÖVE to run headless, but it can be done. Mostly it involves disabling and love.window modules in conf.lua and being careful with the order of loading. You also have to make sure that no calls to functions happen outside your love.draw function.

-1:-- in which a spacefaring text editor holds up against an onslaught (Post Phil Hagelberg)--L0--C0--July 20, 2016 05:48 PM

Bryan Murdock: Another SystemVerilog Streaming Example: Size Mismatch

I had a packed struct who's size was not evenly divisible by 8 (it was one bit short, in fact) and I had an array of bytes that I needed to stream into it. The extra bits in the array of bytes were not relevant, so I tried just doing this:

my_struct = {>>byte{my_array_of_bytes}};

But my simulator complained that my_array_of_bytes was bigger than the destination (my_struct). It took me longer to figure out than I'd like to admit that I just needed to do this:

bit extra_bit;
{my_struct, extra_bit} = {>>byte{my_array_of_bytes}};

That did the trick.
-1:-- Another SystemVerilog Streaming Example: Size Mismatch (Post Bryan ( 19, 2016 04:44 PM

sachachua: 2016-07-18 Emacs News

The post 2016-07-18 Emacs News appeared first on sacha chua :: living an awesome life.

-1:-- 2016-07-18 Emacs News (Post Sacha Chua)--L0--C0--July 18, 2016 03:43 PM

Ivan Kanis: Nano editor easter egg

Here is how to display the easter egg in nano. Run nano and enter some text. Press C-x, press y, enter "zzy" as the file name. I will not say what happens next as it spoils the fun.

As far as I know Emacs does not have one. Well there is M-x butterfly...

-1:-- Nano editor easter egg (Post Ivan Kanis)--L0--C0--July 18, 2016 12:00 AM

Chen Bin (redguardtoo): Enhance emacs-git-gutter with ivy-mode

CREATED: <2016-07-14 Thu>

UPDATED: <2016-07-15 Fri>

emacs-git-gutter show an icon in the gutter area indicating whether a line has been inserted, modified or deleted in Emacs.

I usually use M-x git-gutter:previous-hunk or M-x git-gutter:next-hunk to navigate between the hunks.

But if there are too many hunks in one file, M-x git-gutter:previous-hunk based on ivy-mode is more useful:

(require 'ivy)
(require 'git-gutter)

(defun my-reshape-git-gutter (gutter)
  "Re-shape gutter for `ivy-read'."
  (let* ((linenum-start (aref gutter 3))
         (linenum-end (aref gutter 4))
         (target-line "")
         (target-linenum 1)
         (tmp-line "")
         (max-line-length 0))
      ;; find out the longest stripped line in the gutter
      (while (<= linenum-start linenum-end)
        (goto-line linenum-start)
        (setq tmp-line (replace-regexp-in-string "^[ \t]*" ""
                                                 (buffer-substring (line-beginning-position)
        (when (> (length tmp-line) max-line-length)
          (setq target-linenum linenum-start)
          (setq target-line tmp-line)
          (setq max-line-length (length tmp-line)))

        (setq linenum-start (1+ linenum-start))))
    ;; build (key . linenum-start)
    (cons (format "%s %d: %s"
                  (if (eq 'deleted (aref gutter 1)) "-" "+")
                  target-linenum target-line)

(defun my-goto-git-gutter ()
  (if git-gutter:diffinfos
      (let* ((collection (mapcar 'my-reshape-git-gutter
        (ivy-read "git-gutters:"
                  :action (lambda (linenum)
                            (goto-line linenum))))
    (message "NO git-gutters!")))



-1:-- Enhance emacs-git-gutter with ivy-mode (Post Chen Bin)--L0--C0--July 14, 2016 01:35 PM

Chen Bin (redguardtoo): Turn off linum-mode when file is too big

CREATED: <2016-07-14>

UPDATED: <2016-07-20 Wed>

It's well known that linum-mode slows Emacs when the file contains thousands of lines.

Here is the fix,

(add-hook 'prog-mode-hook
          (lambda ()
            ;; turn off `linum-mode' when there are more than 5000 lines
            (if (and (> (buffer-size)
                        (* 5000 80)))
                     (linum-mode -1))))
-1:-- Turn off linum-mode when file is too big (Post Chen Bin)--L0--C0--July 14, 2016 01:21 PM

Endless Parentheses: A quick guide to directory-local (or project-specific) variables

One of the questions we get most often about CIDER is “can I configure X on a per-project basis?”. Occasionally, you find someone suggesting (or even implementing) some sophisticated configurable variable trying to account for multiple simultaneous use-cases. Fortunately that’s one effort we don’t need to make. Emacs already has that built-in in the form of directory-local variables (dir-local for short).

As the name implies, dir-local variable values (can) apply to all files inside a given directory (also applying recursively to its sub-directories), though you can also restrict them by major-mode or by subdirectory. To configure it:

  1. Invoke M-x add-dir-local-variable.
  2. This will prompt you for a major-mode, then a variable name, and finally a variable value.
  3. Finally, it will drop you in a (newly-created) file called .dir-locals.el in the current directory. Just save this file and you’re done!

By doing this, you have configured the given variable to always have the provided value anywhere inside the current directory, so make sure you’re at the root of your project when you call the command. If you use Projectile, there’s the projectile-edit-dir-locals command for doing just that.

Also worth noting:

  • If you type nil for the major-mode, it applies to all major-modes.
  • There is (of course) tab-completion in the variable-name prompt.
  • If you answer eval for the variable name, it will prompt you for a lisp expression, instead of a value. This expression will be saved and will be evaluated every time a file is visited.
  • After doing this, call revert-buffer on any previously-open files to apply the new value.

Update 21 Jul 2016

It’s worth mentioning that Joel McCracken posted a similar quick guide a few years ago. Some of the information is the same, but some is complementary, so you might want to have a look.

Comment on this.

-1:-- A quick guide to directory-local (or project-specific) variables (Post)--L0--C0--July 05, 2016 12:00 AM

Marcin Borkowski: Compiling a single Beamer frame in AUCTeX

Some time ago I basically stopped using Beamer. I turned to reveal.js, and did a few presentation in it, using Org-mode’s exporter. That was nice, my slides were not “like anyone else’s” (since everyone and their mother uses Beamer), and even equations worked great thanks to MathJax. Some time ago, I taught a Calculus course (which is a challenge on its own, probably worth a blog post or two, especially that it is not aimed at mathematicians, but geographers – but that’s another story). The slides are very diagram-heavy – you know, all sorts of stuff like tangent lines (since we’re doing derivatives), approximations of arc length or areas (since we’re doing integrals), and occasional cute pictures (since we’re also having fun, or so I hope at least). And it turned out that reveal.js (or at least the Org reveal exporter) really doesn’t like images. And it didn’t let me zoom them. Maybe if I were a CSS wizard, I could make it play nice. But I’m not, and I started missing Beamer, with its very precise control of what goes where. So I came back.
-1:-- Compiling a single Beamer frame in AUCTeX (Post)--L0--C0--July 04, 2016 08:44 PM

Ivan Kanis: maybe


climb old man of hoy

write clippy.el

i♥emacs emacs for beginners


The first section is where I capture stuff. My weekly review is a natural process. I start at the top and type 'w' to refile. Using speed keys really help. If you don't know about it check out the documentation in the function org-speed-command-help.

Then I go through each section and see what needs to be done. I tried to be ruthless and put stuff in someday or maybe so that my list is current and feels fresh.

I then finish with things that are in someday (one day I will do it) to maybe (who knows?).

That pretty much sums up how I implement GTD with org mode.

-1:-- maybe (Post Ivan Kanis)--L0--C0--July 04, 2016 03:23 AM

Michael Fogleman: 2014: A Year of Emacs

My Emacs "birthday" is January 21, 2014. That's right-- I just started using Emacs this year! While it's not quite my one-year birthday, I thought I'd take advantage of the annual review blogging tradition to do a retrospective on my first year in Emacs. Note: much of the following chronology takes place during an extended romp through Asia.

I installed Arch Linux in the Fall of 2013. When using Linux, you often need to edit configuration files. The friend who helped me set me up with Vim. I learned the basics--how to insert, save, and quit--and not much else.

Arch on a bench

In January, I decided it might be fun to give Emacs a try. Here is a picture of Marina Bay in Singapore, taken just before or while I did the Emacs tutorial:

Marina Bay in Singapore, just before or during my Emacs birth

As a life-long user of keyboard shortcuts, I appreciated Emacs' notation for key bindings. The tutorial begins with basic cursor controls; easy enough, but nothing exciting. I could see the line, sentence, and buffer equivalents being useful. But my jaw dropped when it taught me C-t, transpose-chars. I knew that mastering just that command would probably save me a lot of pain and typos, and that was probably the least of what Emacs could do.

While the Emacs tutorial probably isn't meant to take more than an hour or two on the first try, it took me probably twice as long. This was because I made Anki cards. Some people say that Emacs' learning curve is steep. For myself, I've found Emacs' learning curve to be gentle, but (pleasantly) endless. Then again, it was probably making Anki cards that made the learning curve less steep for me.

A variation of this card was probably one of my first Emacs cards:

Anki card: transpose-chars 1

Cloze deletion cards like this one are very convenient for this kind of thing. One reason is that it's easy to add new, related information later:

Anki cards transpose-chars 2

In February, the battery on my little Arch laptop died. I spent most of the month without a laptop, waiting for a new battery to arrive by mail. I borrowed my friend and travel partner's machine to review my Anki cards. Still, I did manage to have a little time to develop my Emacs-fu.

In Hanoi, the hotel room we stayed in had a (virus-ridden) Windows machine. What else could I do but install Emacs? At this time, my Emacs configuration was not that complex, so it wasn't worth adapting. As it turned out, that time with a bare Emacs proved especially helpful for mastering some of the key Emacs concepts that I hadn't quite made use of yet; in particular, I got the hang of using multiple buffers in one session. By the the end of February, my new battery had arrived, and I was back in the Emacs saddle.

In March, I converted my my Emacs configuration to an Org-literate format, and started tracking it with Git. This was really the first project that I used Git for, and it gave me an opportunity to learn Git.

The other highlight of March was starting to learn a little Elisp. I found Robert Chassell's "An Introduction to Programming in Emacs Lisp" especially helpful. Chassell's overview and Org-mode's fantastic documentation helped me to write my first significant piece of Elisp (some org-capture templates).

April started off with a bang when my Emacs configuration was featured in a major motion picture, Tron Legacy. But that wasn't nearly as exciting as making my first major mode, tid-mode.

In late June, Artur Malabarba launched his Emacs blog, Endless Parentheses. One of his early posts was about hungry-delete. I was excited about it, but found that it did not work with transient-mark-mode. Artur encouraged me to write a fix. I was very excited when Nathaniel Flath, the maintainer, agreed to merge my patches into hungry-delete. At about the same time, Artur posted an adapted version of my narrow-or-widen-dwim function to Endless Parentheses. Sacha Chua, Nathaniel Flath, and some other Emacs Lisp hackers also offered suggestions.

At the end of July, I started learning Clojure, and added an initial CIDER configuration. While reading the Clojure Cookbook, I decided to hack out a function that turned the pages. Ryan Neufeld, one of the co-authors, merged my pull request, and a while later, I saw my function mentioned on Planet Clojure.

In October, I purchased a Mac for a Clojure contracting job. (Knowing Emacs helped me get the job!) Adding the :ensure keyword to my use-package declarations made the Emacs configuration portion of my OS X set-up process near-instant and pain-free. It was a really cool feeling to be using the same Emacs configuration on two different machines. By the end of the month, both were running on the newly-released Emacs 24.4.

Once Emacs 24.4 was out, Artur posted an Emacs 25 wishlist to Endless Parentheses. I responded with this tweet:

This tweet indirectly led to the happy occasion of the Emacs Hangouts.

That just about brings us to this post. In retrospect, I can't believe it's been less than a year, or that I've learned so much. Thanks to everyone who has helped me so far on the endless path of mastering Emacs, especially Ben, Eric, Sacha Chua, Artur Malabarba, and Bozhidar Batsov.

Joseph Campbell was a Lisp guru

I want to close this post out with Sacha's tweet about 2015:

Like Sacha, I want to learn how to write and run tests in Emacs. I want to learn more about org-mode's features and possible workflows. More broadly, I'd like to make more posts about Emacs, and find other ways to contribute to the Emacs community. I'd like to learn more about Emacs core development. In terms of contrib, I'd like to help accelerate the release of Magit's "next" branch. Above all, I'm sure I'll keep tweaking my configuration.

-1:-- 2014: A Year of Emacs (Post Michael Fogleman)--L0--C0--July 04, 2016 03:18 AM

Michael Fogleman: Emacs can keep in touch

I've wanted to show a specific Emacs workflow that I've established at the Emacs Hangouts that I've attended, but thought it might require too much explanation for an Emacs Hangout. Hence, this post. You might be interested in this workflow if you regularly edit a specific text file in a particular way, run an external program with the same parameters. You might also get some ideas if you're an Emacs-wielding programmer, and wish Emacs had some particular functionality, but would prefer to use Bash or another language to give it that functionality.

Some time ago, my friend Ben started working on a side project called "Keep In Touch." The premise: you have friends, family, and acquaintances that you'd like to stay in touch with. You maintain a list of those people, how often you'd like to contact them, and when you last contacted them. Keep In Touch can then figure out who you are out of touch with. For example, if you want to talk to someone every 30 days, but it's been 31 days, then you are out of touch with them, and the program tells you so. Simple!

Since I started using the program, I've fallen in love with using it. It has helped me to stay in touch with more people, more often. Moreover, I re-implemented Keep in Touch in Clojure as an exercise. And while Keep in Touch provides a serviceable CLI, I've worked out a simple Emacs interface for maintaining the database and viewing the program's output.

It all starts with telling Emacs where it can find the database:

(setq keepintouch-datafile "~/Dropbox/")

Next, I needed a way to tell Emacs that I had contacted someone. The result is an interactive function that asks you who you contacted. It ordinarily assumes that you contacted the person today, but with a prefix argument, it asks you for an alternate date. Then it opens the file and makes the appropriate changes.

(defun keptintouch (arg)
  "Request a contact in a file, and update their last
  contacted date (either today, or, if a prefix is supplied, a user-supplied date.)"
  (interactive "P")
  (let ((contact (read-string "Who did you contact? "))
        (date (if (equal arg nil)
                  (format-time-string "%Y/%m/%d")
                (read-string "When did you contact them? (year/month/date): "))))
      (find-file keepintouch-datafile)
      (goto-char (point-min))
      (search-forward contact)
      (forward-line -1)
      (insert date)
      (switch-to-buffer (other-buffer))
      (kill-buffer (other-buffer)))
    (message "%s was contacted." contact)))

"keptintouch" uses read-string to ask you who contacted, and, if need be, when you contacted them. It uses format-time-string to format today's date in the appropriate format.

The body of the function works almost like a keyboard macro. Often, you can start writing an Elisp function by hitting "C-h k" (describe-key) and finding out what functions are called when you execute the steps as keystrokes. Another way to do this quickly might be to make the keyboard macro, run edit-last-kbd-macro, and use the function annotations to read their documentation or make pseudo-code.

One reason that this is a function instead of a keyboard macro is that it uses some buffer management functions. (All interactive functions are of course accessible to keyboard macros through keyboard strokes, which can really save you some time in one-off situations!) It opens my Keep In Touch database in a new buffer, finds the contact, edits their last contacted date, saves, closes, and switches the buffers. If all went well, it reports success with message. If for some reason the contact is non-existent, search-forward will throw an error. (If that happens, or I want to view or edit the datafile manually, I find it with ido-recentf-open.)

All of this is wrapped in save-excursion, which is kind of like the shell commands popd and pushd except for buffers. In other words, it does something, and then returns the user to what they were doing before they executed the function.

Now that I had a way to maintain the database, I needed a way to run Keep in Touch and view its output, a sorted backlog of people that you are out of touch with.

The Clojure version of Keep in Touch is designed to be exported to a .jar file. Running the resulting program from the command line looks like this:

# note the special location of the .jar
cd target/uberjar
java -jar keepintouch-0.1.0-SNAPSHOT-standalone.jar ~/Dropbox/ schedule backlog

I decided to look at shell-command's documentation to find a way to run this command and put the results into an output buffer. As it turns out, shell-command is that function. As expected, it takes an argument for the desired shell-command. But it also allows for two optional arguments: the name of an output buffer, and the name of an error buffer.

(defun keptintouch-backlog ()
  "Create a buffer with Keep In Touch backlog."
  (let ((src "~/src/keepintouch/clj/keepintouch")
        (jar "-jar target/uberjar/keepintouch-0.1.0-SNAPSHOT-standalone.jar")
        (cur default-directory)) 
    (cd src)
     (concat "java " jar " " keepintouch-datafile " schedule backlog")
     "*Keep In Touch Backlog*")
    (cd cur)))

You'll see that the concat will expand to the shell command that runs the .jar, and that it is followed by an appropriate buffer name.

All of this Lisp makes its way to my finger with two key-binds (by way of use-package's bindkey.el):

(bind-keys ("C-c k" . keptintouch)
           ("C-c K" . keptintouch-backlog))

One variable, two functions, and two keybinds are all it took to make my life easier and get back to staying in touch with friends. You can eat your Emacs cake, and have defun, too.

-1:-- Emacs can keep in touch (Post Michael Fogleman)--L0--C0--July 04, 2016 03:18 AM

William Denton: Tidy data in Org

Today at work I was doing some analysis of spending on electronic resources. I’d done it a few months ago on fiscal year 2015, in a hacky kind of way, but now that F2016 is complete I had two years ago data to work with. As usual I used Org and R, but this time I rejigged everything to use Hadley Wickham’s idea of tidy data and his tools for working with such data, and it made things not only simpler to work with in R but also to present with in Org.

Here’s a simplified example of what it looked like.lo First, I load in the R packages I’ll need for this brief example. In Org hitting Ctrl-c Ctrl-c runs these code blocks. This one is configured to have no output.

#+BEGIN_SRC R :session :results silent :exports none

Next, a table of data: costs of things that librarians spend money on. (We can’t share our eresource spending data … perhaps some day.) This table is meant for people to read and it will appear in the exported PDF. The way it’s presented is good for humans, but not right for machines. I call it tab_costs because it’s a table of costs and I’m going to need to refer to the table later.

#+NAME: tab_costs
| name      | F2015 | F2016 |
| books     |   200 |   300 |
| cardigans |    90 |    60 |
| cats      |   400 |   410 |
| champagne |    80 |    90 |
| internet  |   130 |   140 |
| notebooks |    50 |    60 |
| tea       |    30 |    35 |

The way I have Emacs configured, that looks like this (with extra-prettified source blocks):

Emacs rocks. Emacs rocks.

The next bit of R reads that table into the variable costs_raw, which I then transform with tidyr’s gather function into something more machine-useable. The gather statement says take all the columns except “name” and turn the column names into “year” and the cell values into “cost”. So I can see it and make sure it’ll work, the output is given, but :exports none means that this table won’t be exported when the document is turned into a PDF. Only I can see this, in Emacs.

#+BEGIN_SRC R :session :results values :exports none :var costs_raw=tab_costs :colnames yes
costs <- costs_raw %>% gather(year, cost, -name)

| name      | year  | cost |
| books     | F2015 |  200 |
| cardigans | F2015 |   90 |
| cats      | F2015 |  400 |
| champagne | F2015 |   80 |
| internet  | F2015 |  130 |
| notebooks | F2015 |   50 |
| tea       | F2015 |   30 |
| books     | F2016 |  300 |
| cardigans | F2016 |   60 |
| cats      | F2016 |  410 |
| champagne | F2016 |   90 |
| internet  | F2016 |  140 |
| notebooks | F2016 |   60 |
| tea       | F2016 |   35 |

That’s hard for humans to read, but it means making a chart comparing spending across the two years is easy.

#+BEGIN_SRC R :session :results graphics :exports results :file /tmp/tmp.png :width 600 :height 400
ggplot(costs, aes(x = year, y = cost)) + geom_bar(aes(fill = name), stat = "identity", position = "dodge") + labs(x = "", y = "$", title = "Librarian spending")
What librarians spend money on. What librarians spend money on.

Or (see the geom_bar docs for more):

#+BEGIN_SRC R :session :results graphics :exports results :file /tmp/tmp-year.png :width 600 :height 400
ggplot(costs, aes(x = name, y = cost)) + geom_bar(aes(fill = year), stat = "identity", position = "dodge") + labs(x = "", y = "$", title = "Librarian spending")
Another way to show. Another way to show.

Another Emacs screenshot showing how Org mixes code, graphics and text (well, text if I’d written some, but I didn’t here):

Emacs rocks! Emacs rocks!
-1:-- Tidy data in Org (Post William Denton)--L0--C0--July 01, 2016 03:02 AM

Grant Rettke: KEYR8 (Emacs Friendly Keyboard): Prototype Fabricated


They took a newbie like me through the process of designing the thing and it was fun. Then they fabricated the thing and it rocks.

Emacs friends: there is enough key-space (128 keys) to easily place

  • Control
  • Meta
  • Super
  • Hyper
  • Ultra (Control-Meta-Super-Hyper)
  • Alt
  • Gui


Strong thumb proponents: I’m trying to make it happen here by placing everything on the bottom two rows (including space, Fn, and enter).


-1:-- KEYR8 (Emacs Friendly Keyboard): Prototype Fabricated (Post Grant)--L0--C0--June 29, 2016 01:50 AM

Endless Parentheses: A few paredit keys that take over the world

Once you learn paredit, you really can’t go back. Even when you’re not editing lisp you crave for the ease of manipulating everything as balanced sexps. Although that’s (sadly) not always possible, you can still hack your way into a bit of guilty paren-pleasure in pretty much any editing session.

While not all programming languages are lisps, most of them do have brackets and quotes. And paredit should have no problem moving you forward-up out of a C string, wrapping curly brackets around a work in LaTeX, or even splicing out a pair of parenthesis in plain prose.

Below are a few keys I find useful pretty much everywhere, so I’ve allowed them to take over the global keymap.

(global-set-key (kbd "C-M-u") #'paredit-backward-up)
(global-set-key (kbd "C-M-n") #'paredit-forward-up)
;; This one's surpisingly useful for writing prose.
(global-set-key "\M-S"
(global-set-key "\M-R" #'paredit-raise-sexp)
(global-set-key "\M-(" #'paredit-wrap-round)
(global-set-key "\M-[" #'paredit-wrap-square)
(global-set-key "\M-{" #'paredit-wrap-curly)

Comment on this.

-1:-- A few paredit keys that take over the world (Post)--L0--C0--June 29, 2016 12:00 AM

(or emacs: Bookmark the current window layout with Ivy

Today's post is about the newest feature related to the ivy-switch-buffer command. If you use ivy-mode, you're probably already using ivy-switch-buffer since it overwrites the built-in switch-to-buffer.

The cool thing about ivy-switch-buffer is that it's not only buffers that are offered for completion. Other buffer-like entities can be there as well: bookmarks, recently opened files (recentf), and finally, window layouts. Since all of those are relatively the same concept, it's very convenient to have them all in one place available for completion.

Here are the relevant settings:

;; Enable bookmarks and recentf
(setq ivy-use-virtual-buffers t)

;; Example setting for ivy-views
(setq ivy-views
      `(("dutch + notes {}"
          (file "")
          (buffer "notes")))
        ("ivy.el {}"
          (file ,(find-library-name "ivy"))
          (buffer "*scratch*")))))

I did mention ivy-views before in the ivy-0.8.0 release post. But now, instead of setting ivy-views by hand, you can also bind ivy-push-view to a key and store as many window configurations as you like, really fast.

What gets stored:

  • The window list - all windows open on the current frame.
  • The window splits relative to each other as a tree. Currently, the size of the split isn't saved, all windows are split equally.
  • The point positions in each window. If you use just one window, you've got something similar to bookmark-set.

Here's what I use currently:

(global-set-key (kbd "C-c v") 'ivy-push-view)
(global-set-key (kbd "C-c V") 'ivy-pop-view)

Typical workflow

Suppose I have two files open: the file and the _posts directory. By pressing C-c v I am prompted for a view name with the default being e.g. {} _posts 2.

I can still name the view however I want, but I typically just press RET. The parts of the automatic view name are:

  • {} - this is a simple string marker to distinguish the views in the buffer view. If I enter only {} into ivy-switch-buffer prompt, the candidates will normally filter to only views, since very rarely will a file or a buffer name match {}.
  • _posts is the list of buffers stored in the view. This view has only two buffers, but ivy-push-view can handle as many windows as you can cram into a single frame.
  • 2 means that I already have two views with the same buffers, each new view with the same buffers gets an increased number for the suggested name. And it's not useless to have many views for the same buffers, since the views also store point positions, not just the window list.

Here's the beauty of it for me: when I type _posts into ivy-switch-buffer I can chose to open the _posts directory in a variety of ways:

  • If the buffer is currently open, I can just switch there.
  • If the buffer is currently closed, I can re-open it, thanks to recentf.
  • I can open the buffer as part of a stored view(s) in ivy-views.

Finally, if I decide that I don't need a particular view any more, I can delete it with C-c V (ivy-pop-view). It's possible to delete many views at once by pressing C-M-m (ivy-call), as usual with most ivy completion functions.

Breaking API change

While implementing ivy-set-view I decided that the current way alist collections are handled together with actions is sub-optimal. Here's the new way of working:

(let (res)
   '(ivy-read "test: "
     '(("one" . 1) ("three" . 3))
     :action (lambda (x) (setq res x)))
   "t C-m")
;; =>
;; ("three" . 3)

Previously, the return result would be 3, i.e. the cdr of the selected candidate. Any code using ivy-read with an alist-type collection will break. I fixed all instances in counsel.el, and there actually aren't too many uses in the published third party packages.

A simple fix to the problem is to use cdr in the action function. Additionally, having more information available in the action function will serve to improve the code.

-1:-- Bookmark the current window layout with Ivy (Post)--L0--C0--June 26, 2016 10:00 PM

Ben Simon: Cutting the electronic cord: Setting up a fully paper TODO list tracking strategy

A few months back I started a new TODO list strategy. Rather than having a master task list in the Cloud and creating a daily TODO list in a paper notebook, I maintained both an electronic and paper master task list. The electronic version was tracked in git + emacs + org-mode and the paper version was on index cards.

While git + emacs + org-mode was certainly functional, I never had cause to do anything particular sexy with the setup. In fact, I was hoping this experience would convert me from a fan of subversion to a fan of git, but alas, it only reinforced my appreciation for the simplicity of subversion.

The index cards, on the other hand, were a joy to use. I love that each project is represented by a single card, and that spreading out the cards gives me an overview of all possible tasks to work on:

My daily ritual has become this: brew a cup of tea, spread out the cards, and review where I'm at. I then choose a sequence of tasks to tackle for the day, and stack the cards accordingly:

As I complete tasks, I cross out the line item in question. Switching projects means putting the card on top at the back of the deck, and giving my full attention to newly visible card.

The 10 lines of a 3x5 index card are perfect for keeping tabs on active tasks on a project. If all goes well, project cards become a crossed out mess. No biggie, these cards get recycled (for example: as drawing canvases), and I create a fresh card for the project. The turn over in cards helps keep both physical and metal clutter to a mininum.

I have three recurring activities that I like to fit into my day: scrubbing my work e-mail, scrubbing my personal e-mail and blogging. I wrapped these cards in packing tape, to make them more durable. As a bonus, they serve as tiny whiteboards. These special cards get integrated into the daily stack like any other project.

A few weeks back I splurged on a set of colored pens, and the result is that I can now color code information on the cards with ease.

There's no doubt that part of what I enjoy about this system is that the physical actions on the card reinforce my mental goals. For example, when I sequence the cards for the day, put them in a stack, and attach a mini-binder clip, I'm reinforcing the change-over from thinking big picture to thinking only about a specific task.

So the setup works. Of late, however, the drag of maintaining tasks both electronically and in paper form was getting to me. Yes, updating a task in both places takes just a few seconds, but still, all those seconds add up. So it was time to cut the cord and either go all electronic or all paper. Given the benefits of the paper strategy, I decided to go that route.

Before switching strictly to paper, however, I needed to account for the two main benefits that the electronic system was providing. These include: the always-available always-backed-up nature of storing a text file in git, and the quick linking capabilities offered by emacs + org-mode.

I have pretty strict rule about my task list: never depend on my memory. Ever. If I were to get bonked on the head and suffer from short-term amnesia, I should be able to look at my task list and know exactly what I should work on next. So yeah, I take the integrity of my task list very seriously. Depending on a set of index cards which could be lost, forgotten in a coffee shop, run through the washing machine or destroyed in a freak tea spilling, is a bad idea. In short, I needed a backup strategy.

Turns out, this was an easy conundrum to solve. Every morning I spread out the cards to see what I should work on that day. The solution: I snap a photo of these spread out cards. Problem solved. The photo is backed up in the cloud and accessible everywhere. Yes, it means I have a daily backup and not the every-single-change backup that git provides, but I can live with that. As a bonus, it forces me to not get lazy and skip the planning-overview step of my process. Problem #1, solved.

The second challenge has to do with linking tasks to more information. As I said above, I don't like to depend on my memory. Another manifestation of this principle is that when I create a TODO item I like to link it back to a detailed source of informative. Consider this fake task list:

Project Xin-Gap
* Fix user login issue

When I documented this as a task, I knew exactly what "login issue" I was referring to. Two weeks later (or one big bonk on the head), I may have no clue. org-mode makes it very easy to link items in the outline to a URL. In the above case, I'd link the text "Fix user login issue" to either a bug report URL or to the URL of the e-mail message where the issue was reported. These links allow my TODO list to remain a sort of tip of the iceberg: the details, like the majority of the iceberg, are hidden from view. But they're there.

So how do I replicate these links in an index card environment? This one took a little longer to figure out. Possible ideas included: NFC stickers and QR Codes. Ultimately, I realized what I needed was to leverage a URL shortener.

For example, if I wanted to link to a bug report over at:, I could drop that URL into and get out something like I could then note 28UgMwR on the index card. To access the bug report, I reverse the process: use the code to form a link, which in turn will take me to the bug report. The problem is, manually running these steps for was too time consuming and out of the question.

After experimenting a bit with YOURLS, I finally settled on an even simpler approach. I already have a URL shortener setup for my Google Apps account. It was a Google Labs Project back in the day, and somewhat shockingly, it still runs without issue. I access it via: I found that I if visit the URL:

I'm taken to this page:

I can then write Z23 on my Index Card and hit Add Short Link on the above screen, and I'm all set. Google's URL shotener even comes with a Firefox Protocol Handler which means that I can type: goto:z23 and the browser will expand the URL and visit the correct page.

To streamline this process I created a bookmarklet that does almost all of this for me:

(function() {
  function c() {
    var dict = ['A','B','C','D','E','F','H','J','K','L','M','N',
    return dict[Math.floor(Math.random() * dict.length)];
  }'' +
              'url='+encodeURIComponent(location.href) + '&' +
              'path=' + c() + c() + c(), '_blank');

With one click on my toolbar, I get a browser tab opened to the add-short-link page where there's a random 3 digit code (minus any ambiguous characters like l vs 1 or 0 vs O). I note the 3 digit code on paper, click Add Short Link and close the tab. When I want to visit a task's linked item, I just enter the URL goto:ZZZ where ZZZ is the 3 digit code that I've noted.

Of course, if you want to pursue this, you'll need to setup your own link shortener. And for the life of me, I can't find any indication as to how you'd setup the Google Labs URL shortener I'm making use. But you'll figure it out, I'm sure. The bottom line is, you can combine a URL Shotener and a bookmarklet to make paper linking relatively painless.

With my two challenges solved, I went ahead and cut the electronic cord. It feels strange to depend solely on a stack of cards that I could trivially lose. But my daily photo backup gives me confidence, and the daily handling of cards reminds me that there's real power in working with physical materials.

Besides, it's only a matter of time before I refine my system again. As long as I have TODO items, I'll be on the lookout for a better way of organizing them.

-1:-- Cutting the electronic cord: Setting up a fully paper TODO list tracking strategy (Post Ben Simon ( 23, 2016 05:50 PM

Raimon Grau: TIL: Toggle tracing defuns with slime

A nice and quick way to trace/untrace defuns from slime:

(define-key slime-mode-map (kbd "C-c t") 'slime-toggle-trace-fdefinition)
-1:-- TIL: Toggle tracing defuns with slime (Post Raimon Grau ( 21, 2016 11:06 AM

Chris Wellons: Elfeed, cURL, and You

This morning I pushed out an important update to Elfeed, my web feed reader for Emacs. The update should be available in MELPA by the time you read this. Elfeed now has support for fetching feeds using a cURL through a curl inferior process. You’ll need the program in your PATH or configured through elfeed-curl-program-name.

I’ve been using it for a couple of days now, but, while I work out the remaining kinks, it’s disabled by default. So in addition to having cURL installed, you’ll need to set elfeed-use-curl to non-nil. Sometime soon it will be enabled by default whenever cURL is available. The original url-retrieve fetcher will remain in place for time time being. However, cURL may become a requirement someday.

Fetching with a curl inferior process has some huge advantages.

It’s much faster

The most obvious change is that you should experience a huge speedup on updates and better responsiveness during updates after the first cURL run. There are important two reasons:

Asynchronous DNS and TCP: Emacs 24 and earlier performs DNS queries synchronously even for asynchronous network processes. This is being fixed on some platforms (including Linux) in Emacs 25, but now we don’t have to wait.

On Windows it’s even worse: the TCP connection is also established synchronously. This is especially bad when fetching relatively small items such as feeds, because the DNS look-up and TCP handshake dominate the overall fetch time. It essentially makes the whole process synchronous.

Conditional GET: HTTP has two mechanism to avoid transmitting information that a client has previously fetched. One is the Last-Modified header delivered by the server with the content. When querying again later, the client echos the date back like a token in the If-Modified-Since header.

The second is the “entity tag,” an arbitrary server-selected token associated with each version of the content. The server delivers it along with the content in the ETag header, and the client hands it back later in the If-None-Match header, sort of like a cookie.

This is highly valuable for feeds because, unless the feed is particularly active, most of the time the feed hasn’t been updated since the last query. This avoids sending anything other hand a handful of headers each way. In Elfeed’s case, it means it doesn’t have to parse the same XML over and over again.

Both of these being outside of cURL’s scope, Elfeed has to manage conditional GET itself. I had no control over the HTTP headers until now, so I couldn’t take advantage of it. Emacs’ url-retrieve function allows for sending custom headers through dynamically binding url-request-extra-headers, but this isn’t available when calling url-queue-retrieve since the request itself is created asynchronously.

Both the ETag and Last-Modified values are stored in the database and persist across sessions. This is the reason the full speedup isn’t realized until the second fetch. The initial cURL fetch doesn’t have these values.

Fewer bugs

As mentioned previously, Emacs has a built-in URL retrieval library called url. The central function is url-retrieve which asynchronously fetches the content at an arbitrary URL (usually HTTP) and delivers the buffer and status to a callback when it’s ready. There’s also a queue front-end for it, url-queue-retrieve which limits the number of parallel connections. Elfeed hands this function a pile of feed URLs all at once and it fetches them N at a time.

Unfortunately both these functions are incredibly buggy. It’s been a thorn in my side for years.

Here’s what the interface looks like for both:


It takes a URL and a callback. Seeing this, the sane, unsurprising expectation is the callback will be invoked exactly once for time url-retrieve was called. In any case where the request fails, it should report it through the callback. This is not the case. The callback may be invoked any number of times, including zero.

In this example, suppose you have a webserver on port 8080 that will return an HTTP 404 at the given URL. Below, I fire off 10 asynchronous requests in a row.

(defvar results ())
(dotimes (i 10)
  (url-retrieve ""
                (lambda (status) (push (cons i status) results))))

What would you guess is the length of results? It’s initially 0 before any requests complete and over time (a very short time) I would expect this to top out at 10. On Emacs 24, here’s the real answer:

(length results)
;; => 46

The same error is reported multiple times to the callback. At least the pattern is obvious.

(cl-count 0 results :key #'car)
;; => 9
(cl-count 1 results :key #'car)
;; => 8
(cl-count 2 results :key #'car)
;; => 7

(cl-count 9 results :key #'car)
;; => 1

Here’s another one, this time to the non-existent foo.example. The DNS query should never resolve.

(setf results ())
(dotimes (i 10)
  (url-retrieve "http://foo.example/"
                (lambda (status) (push (cons i status) results))))

What’s the length of results? This time it’s zero. Remember how DNS is synchronous? Because of this, DNS failures are reported synchronously as a signaled error. This gets a lot worse with url-queue-retrieve. Since the request is put off until later, DNS doesn’t fail until later, and you get neither a callback nor an error signal. This also puts the queue in a bad state and necessitated elfeed-unjam for manually clear it. This one should get fixed in Emacs 25 when DNS is asynchronous.

This last one assumes you don’t have anything listening on port 57432 (pulled out of nowhere) so that the connection fails.

(setf results ())
(dotimes (i 10)
  (url-retrieve ""
                (lambda (status) (push (cons i status) results))))

On Linux, we finally get the sane result of 10. However, on Windows, it’s zero. The synchronous TCP connection will fail, signaling an error just like DNS failures. Not only is it broken, it’s broken in different ways on different platforms.

There are many more cases of callback weirdness which depend on the connection and HTTP session being in various states when thing go awry. These were just the easiest to demonstrate. By using cURL, I get to bypass this mess.

No more GnuTLS issues

At compile time, Emacs can optionally be linked against GnuTLS, giving it robust TLS support so long as the shared library is available. url-retrieve uses this for fetching HTTPS content. Unfortunately, this library is noisy and will occasionally echo non-informational messages in the minibuffer and in *Messages* that cannot be suppressed.

When not linked against GnuTLS, Emacs will instead run the GnuTLS command line program as an inferior process, just like Elfeed now does with cURL. Unfortunately this interface is very slow and frequently fails, basically preventing Elfeed from fetching HTTPS feeds. I suspect it’s in part due to an improper coding-system-for-read.

cURL handles all the TLS negotation itself, so both these problems disappear. The compile-time configuration doesn’t matter.

Windows is now supported

Emacs’ Windows networking code is so unstable, even in Emacs 25, that I couldn’t make any practical use of Elfeed on that platform. Even the Cygwin emacs-w32 version couldn’t cut it. It hard crashes Emacs every time I’ve tried to fetch feeds. Fortunately the inferior process code is a whole lot more stable, meaning fetching with cURL works great. As of today, you can now use Elfeed on Windows. The biggest obstable is getting cURL installed and configured.

Interface changes

With cURL, obviously the values of url-queue-timeout and url-queue-parallel-processes no longer have any meaning to Elfeed. If you set these for yourself, you should instead call the functions elfeed-set-timeout and elfeed-set-max-connections, which will do the appropriate thing depending on the value of elfeed-use-curl. Each also comes with a getter so you can query the current value.

The deprecated elfeed-max-connections has been removed.

Feed objects now have meta tags :etag, :last-modified, and :canonical-url. The latter can identify feeds that have been moved, though it needs a real UI.

See any bugs?

If you use Elfeed, grab the current update and give the cURL fetcher a shot. Please open a ticket if you find problems. Be sure to report your Emacs version, operating system, and cURL version.

As of this writing there’s just one thing missing compared to url-queue: connection reuse. cURL supports it, so I just need to code it up.

-1:-- Elfeed, cURL, and You (Post)--L0--C0--June 16, 2016 06:22 PM

Ben Simon: Got Data? Adventures in Virtual Crystal Ball Creation

There's two ways to look at this recent Programming Praxis exercise: implementing a beginner level statistics function or creating a magical crystal ball that can predict past, present and future! I chose to approach this problem with the mindset of the latter. Let's make some magic!

The algorithm we're tackling is linear regression. I managed to skip statistics in college (what a shame!), so I don't recall ever being formally taught this technique. Very roughly, if you have the right set of data, you can plot a line through it. You can then use this line to predict values not in the data set.

The exercise gave us this tiny, manufactured, data set:

x    y
60   3.1
61   3.6
62   3.8
63   4.0
65   4.1

With linear regression you can answer questions like: what will be the associated value for say, 64, 66 or 1024 be? Here's my implementation in action:

A few words about the screenshot above. You'll notice that I'm converting my data from a simple list to a generator. A generator in this case is a function that will return a single element in the data set, and returns '() when all the data has been exhausted. I chose to use a generator over a simple list because I wanted to allow this solution to scale to large data sets.

Below you'll see a data set that's stored in a file and leverages a file based generator to access its contents. So far, I haven't throw a large data set at this program, but I believe it should scale without issue.

The call to make-crystal-ball performs the linear-regression and returns back a function that when provided x returns a guess prediction for y. What? I'm trying to have a bit of fun here.

Looking around on web I found this example that uses a small, but real data set. In this case, it compares High School and College GPA. Using linear-regression we're able to predict how a High School student with a 2.7, 3.0 or 3.5 GPA is going to do in college. Here's the code:

(define (gpa-sat-test)
  (define data (file->generator "gpa-sat-data.scm"
                                (lambda (high-gpa math-sat verb-sat comp-gpa univ-gpa)
                                  (list high-gpa univ-gpa))))
  (let ((ball (make-crystal-ball data)))
    (show "2.7 => " (ball 2.7))
    (show "3.0 => " (ball 3))
    (show "3.5 => " (ball 3.5))))

And the answer is: 2.91, 3.12 and 3.45 respectively. So yeah, that's good news if you were dragging in High School, you're GPA should climb a bit. But High School overachievers should beware, your GPA is most likely to dip. D'oh.

Below is my implementation of this solution. You can also find it on github. As usual I find myself preaching the benefits of Scheme. The code below was written on my Galaxy Note 5 using Termux, emacs and Tinyscheme. With relative ease I was able to implement a generator framework that works for both lists and data files. I'm also able to leverage the Scheme reader so that the data file format is trivial to operate on. Finally, I wrote a generic sigma function that walks through my data set once, but performs all the various summations I'll need to calculate the necessary values. In other words, I feel like I've got an elegant solution using little more than lists, lambda functions and sexprs. It's beautiful and should be memory efficient.

Here's the code:


(define (show . args)
  (for-each (lambda (arg)
       (display arg)
       (display " "))

(define (as-list x)
  (if (list? x) x (list x)))

(define (g list index)
  (list-ref list index))

(define (make-list n seed)
  (if (= n 0) '()
      (cons seed (make-list (- n 1) seed))))

(define (list->generator lst)
  (let ((remaining lst))
    (lambda ()
      (cond ((null? remaining) '())
      (let ((x (car remaining)))
        (set! remaining (cdr remaining))

(define (file->generator path scrubber)
  (let ((port (open-input-file path)))
    (lambda ()
      (let ((next (read port)))
        (if (eof-object? next) '() (apply scrubber next))))))

(define (sigma generator . fns)
  (define (update fns sums data)
    (let loop ((fns fns)
        (sums sums)
        (results '()))
      (cond ((null? fns) (reverse results))
      (let ((fn (car fns))
     (a  (car sums)))
        (loop (cdr fns)
       (cdr sums)
       (cons (+ a (apply fn (as-list data)))

    (let loop ((data (generator))
        (sums (make-list (length fns) 0)))
    (if (null? data) sums
 (loop (generator)
       (update fns sums data)))))

;; Magic happens here:
;; m = (n × Σxy − Σx × Σy) ÷ (n × Σx2 − (Σx)2)
;; b = (Σy − m × Σx) ÷ n
(define (linear-regression data)
  (let ((sums (sigma data
        (lambda (x y) (* x y))
        (lambda (x y) x)
        (lambda (x y) y)
        (lambda (x y) (* x x))
        (lambda (x y) 1))))
    (let* ((Sxy (g sums 0))
    (Sx  (g sums 1))
    (Sy  (g sums 2))
    (Sxx (g sums 3))
    (n   (g sums 4)))
      (let* ((m (/ (- (* n Sxy) (* Sx Sy))
     (- (* n Sxx) (* Sx Sx))))
      (b (/ (- Sy (* m Sx)) n)))
 (cons m b)))))

(define (make-crystal-ball data)
  (let* ((lr (linear-regression data))
         (m  (car lr))
         (b  (cdr lr)))
    (lambda (x)
      (+ (* m x) b))))

;; Playtime
(define (test)
  (define data (list->generator '((60   3.1)
      (61   3.6)
      (62   3.8)
      (63   4.0)
      (65   4.1))))
  (let ((ball (make-crystal-ball data)))
    (show (ball 64))
    (show (ball 66))
    (show (ball 1024))))

;; From:
(define (gpa-sat-test)
  (define data (file->generator "gpa-sat-data.scm"
                                (lambda (high-gpa math-sat verb-sat comp-gpa univ-gpa)
                                  (list high-gpa univ-gpa))))
  (let ((ball (make-crystal-ball data)))
    (show "2.7 => " (ball 2.7))
    (show "3.0 => " (ball 3))
    (show "3.5 => " (ball 3.5))))


-1:-- Got Data? Adventures in Virtual Crystal Ball Creation (Post Ben Simon ( 16, 2016 04:35 PM

Phillip Lord: Robot Testing With Emacs

I have written about assess previously [1]; it is a tool which provides predicates, macros and functions to support testing for Emacs. It is actually agnostic to the test environment, although has specialised support for ERT.

My new release of assess (v0.3.2) includes one significant change, and two new features. I have updated the call capture functionality — the first version stored all the call data in a global variable, which was quick and easy, but clearly not a log term solution. It now uses closures instead which means that several functions can be captured at once. This also allows the first new feature, which is the ability to capture calls to hooks, with the function assess-call-capture-hook, which takes a hook and a lambda, and returns any calls to the hook when the lambda is evaluated. As an example usage, from assess-call-tests.el:

  '(nil nil)
   (lambda ()
     (run-hooks 'assess-call-test-hook)
     (run-hooks 'assess-call-test-hook)))))

This is good functionality and should be very useful. The API could be improved a bit; a macro version would avoid the explicit lambda, for example. And returning a list of nil means this function also works with hooks with args, but is a bit ugly for hooks without (which are the majority).

The second area that I wanted to address has come about because of my hacking into the Emacs undo system. This is hard to test automatically; I have often found myself writing things like this test from simple-test.el.

   (setq buffer-undo-list nil)
   (insert "hello")
   (member (current-buffer) undo-auto--undoably-changed-buffers)))

This is okay, but it’s painful to write; I am trying to robotize Emacs, and it’s not easy. Some times it’s hard to work out exactly what set of functions you need to call. It would be much easier just to type a key sequence and have Emacs run this for you.

Fortunately, Emacs has special support for this in the form of keyboard macros; you can remember, store and save any set of keypresses and run them, rerun them, automate them or, most importantly, save them to a file as a lisp expression. This example, for instance, comes from viper-test.el.

(kmacro-call-macro nil nil nil
                    ;; Delete "c"
                    left left left
                    ;; Delete "a"
                    ;; C-/ or undo

This is okay, but it’s still not ideal. I have had to add comments to make the test clear by hand. It’s not easy to read, and what is that 67108911 about? It comes from somewhere in Emacs and is stable into the future. But, you only have my word for it that this is undo. It would be all too easy to get this wrong, to have the wrong comment. Tests need to be readable.

Fortunately, Emacs provides a nice solution, in the form of edmacro — this is a major-mode for editing macros after they have been created. It also defines a human readable version of a macro. We can parse this and then execute it directly. This example comes from simple-test.el.

C-c C-o                 ;; latex-insert-block
RET                     ;; newline
C-/                     ;; undo

The advantage of this is that I didn’t actually write this string; I recorded the macro, the edited it and copied the contents of the edmacro buffer.

This is still not easy enough, though; I want an easier way of editing the macro as it appears in the test. This is, unfortunately, difficult as the edit-kbd-macro is not easy to invoke programmatically — it absolutely hard-codes user interaction (I did even try invoking edit-kbd-macro using a keyboard macro!). So, I have given up with that approach in the short term. Instead, I have written a function assess-robot-execute-macro that combines read-kbd-macro and execute-kbd-macro, but also sets the macro as the last macro, making it easy to edit. I’ve also added a keybinding to edmacro to copy the macro to the kill-ring. And here is a test using it:

hello                   ;; self-insert-command * 5

This also demonstrates the final quirk. Keyboard macros work in which ever buffer is selected — not the one which is current. We cannot use with-temp-buffer to select on temporarily and run the macro in it. So I have added macros to display a buffer temporarily instead.

As with many parts of assess, the back end is quite convoluted and complex, as many parts of Emacs were not written with testing in mind (that is they predate the whole idea of unit testing by many years!). But, I hope that the API that assess provides is simple, clear and flexible.

Assess is available at github and MELPA.

Feedback is, as always, welcome.


  1. P. Lord, "Emacs Testing with Assess", An Exercise in Irrelevance, 2016.
-1:-- Robot Testing With Emacs (Post Phillip Lord)--L0--C0--June 13, 2016 02:07 PM