sachachua: 2017-01-16 Emacs News

Links from, /r/orgmode, /r/spacemacs, Hacker News,, Youtube, the changes to the Emacs NEWS file, and emacs-devel.

Past Emacs News round-ups

-1:-- 2017-01-16 Emacs News (Post Sacha Chua)--L0--C0--January 16, 2017 06:09 AM

Irreal: Recurring Org Events

Karl Voit has published an instructive post on how to specify recurring events in Org mode. As most of you know, you can annotate an Org TODO entry to specify that it should occur every day, every week, every 3rd day and other more complex criteria.

Even if you’re familiar with the above, Voit has an alternative method that I wasn’t aware of. That method is org-clone-subtree-with-timeshift that will duplicate the current entry but apply different time stamps. This has some benefits over the more common method. In the first place, if you have a meeting on Tuesday for the next 5 weeks, you can make the first TODO entry and then call org-clone-subtree-with-timeshift with a repeat of 4 and a timeshift of 1w. That gives you the exact number of entries and can edit or delete one or more as necessary. Another benefit is that you can add notes to each entry separately rather than having everything together in a single entry.

Take a look at Voit’s post for the details and more advantages of this method. It’s also a nice summary of how you can specify dates and times whether or not they’re recurring.

-1:-- Recurring Org Events (Post jcs)--L0--C0--January 15, 2017 06:11 PM

Marcin Borkowski: I'm now using the right dictionary

Almost one and half years ago I read an interesting post on Irreal, and – as a great fan of the English language – I immediately went to the original post it mentioned. I did not regret it; I return to that post from time to time, since it is so beautiful, even poetic.
-1:-- I'm now using the right dictionary (Post)--L0--C0--January 14, 2017 07:55 PM

Irreal: A Reminder about Emacs Keybindings on MacOS

Recently, I’ve seen tweets such as this one:

It’s been a few years since I last wrote about that so here, as a public service, is a link to my post that explains how to install Emacs keybindings to MacOS apps. The process is simple so if you’re a heavy Emacs user on a Mac be sure to check it out. I use this and would be lost without it.

-1:-- A Reminder about Emacs Keybindings on MacOS (Post jcs)--L0--C0--January 14, 2017 06:11 PM

sachachua: 2017-01-09 Emacs News

Links from, /r/orgmode, /r/spacemacs, Hacker News,, Youtube, the changes to the Emacs NEWS file, and emacs-devel.

Past Emacs News round-ups

-1:-- 2017-01-09 Emacs News (Post Sacha Chua)--L0--C0--January 10, 2017 07:15 PM

William Denton: Recent Emacs changes

I’ve tweaked my Emacs setup a few times recently.

First, I changed how Org bullets look. I use the org-bullets package—it’s very nice—and had it showing bullets and circles of various sizes for the different heading levels, but a few weeks ago something broke for a little while and I wasn’t seeing any bullets on any headings. Do you know, it looked nice. Very simple and clean. When the problem was fixed I tried it for real, but I need some kind of visual indicator of where I am, so I changed the bullets for the top three headings and use nothing for the fourth. This is what Org looks like for me now:

Org bullets. Org bullets.

The top heading is “⊢,” which is a turnstile or RIGHT TACK in Unicode.

It flows nicely as an outline:

Org bullets outline. Org bullets outline.

Second, I started using Projectile. It’s very nice for working on projects and bouncing around files in one project, and it does a great job of understanding what a project is. C-c p p lets me switch to a project (usually something under version control, but it doesn’t have to be), and then C-c p f finds a file in that project or C-c p b bounces to an open buffer from that project.

Third, I started using Ivy, Swiper and Counsel for command completion and browsing search results. For some reason I had to run M-x package-refresh and M-x package-install RET ivy to get Ivy to go in; perhaps that’s a transient problem. Aside from that it was all seamless and it’s giving me great improvements on searching for words (with C-s), using ag to search files, searching the Unicode table to find a particular character, and so on. It’ll take me a while to learn everything it can do, but out of the box it’s an improvement right away.

Here’s what a search for “zotero” (C-s zotero) looked like while editing a this file a few minutes ago. The matches are highlighted on screen because I use the highlight package, but they’re also there for my choosing in the minibuffer at the bottom thanks to Swiper. It can also show me results from multiple files, and jumping to the line I want is easy.

'C-s zotero' in this buffer: nice results at bottom. 'C-s zotero' in this buffer: nice results at bottom.

Fourth, I’m using the zotxt extension in Firefox so Emacs can talk to Zotero with the zotxt package. Michael Behr wrote this up in A research workflow with Zotero and Org mode and I was very happy to see it, because it does just what I need: inserting citations from Zotero into Org buffers. I keep notes on papers I read, and this does the trick. I’ll go to org-ref (John Kitchin’s work with Org is remarkable) if I need to, but Zotero is my research management tool, and I want to talk to it directly, without exporting BibTeX files from it as a middle step.

-1:-- Recent Emacs changes (Post William Denton)--L0--C0--January 10, 2017 02:20 AM

Pragmatic Emacs: How I view my google calendar agenda in Emacs

JCS over on Irreal describes a couple of ways of syncing a google calendar with your org-mode agenda. I prefer to keep my google calendar separate from my agenda, as I use the latter to track tasks rather than appointments. I still want to see my google calendar agenda in Emacs though so I have a fairly simple work around.

I use gcalcli to access my google calendar from the command line, and run it as a cron job using gcalcli agenda and redirecting the output to a file. Then in Emacs I have a simple function to display the contents of that file.

Firstly though, the output of gcalcli contains ansi colour codes to colour-code the calendars in the agenda. Emacs won’t display these colours unless we tell it to. We use the following code to define a function display-ansi-colors to do this for us (source):

;; define function to display ansi colours for a buffer
(require 'ansi-color)
(defun display-ansi-colors ()
  (ansi-color-apply-on-region (point-min) (point-max)))

Now, here is the code to display the agenda:

(defun bjm/open-gcal-agenda ()
  "Open my google calendar agenda file. The agenda is displayed in the buffer *gcal*."
  ;; set name of calendar buffer and location of file containing my agenda
  (let ((tmp-buff-name "*gcal*") (cal-file (expand-file-name "/homeb/bjm/tmp/gcal")))
    ;; switch to calendar buffer
    (switch-to-buffer tmp-buff-name)
    ;; turn off read only to overwrite if buffer exists
    (read-only-mode -1)
    ;; clear buffer
    ;; insert agenda file
    (insert-file-contents cal-file)
    ;; turn on colours
    ;; turn on special mode
    ;; turn off line wrapping
    (visual-line-mode -1)))

There are a couple of points worth noting here…

Firstly, I could easily just call gcalcli on a timer in Emacs, but I use the agenda file for other things so it is helpful to do that externally.

Secondly, after inserting the contents of the agenda file to the buffer I set it to special mode, which is basically a read only mode with some common key bindings, and is useful for buffers that are not associated with editing files. Related to this, I call the buffer *gcal*, which follows the convention that buffers not associated with files have names starting and ending with *.

Now by calling my bjm/open-gcal-agenda function, I see something like this:


-1:-- How I view my google calendar agenda in Emacs (Post Ben Maughan)--L0--C0--January 09, 2017 12:02 AM

Pragmatic Emacs: Use your digits and a personal key map for super shortcuts

Lots of commands in Emacs behave differently if called with a prefix argument C-u. For example, by default C-s runs the command isearch which searches for text as you type. If you use a prefix argument C-u C-s then isearch searches with regular expressions instead of literal string text.

You can also pass numerical arguments to commands, which for many simple commands will cause the command to repeat that number of times, so for example C-u 3 C-p passes the argument 3 to the command C-p which then runs previous-line three times. What is more, you can also pass these numerical arguments with the form C-3 C-p, where C-3 is equivalent to C-u 3. Of course this works for any digit 0-9, not just 3!

By default, Emacs uses the combinations C-3, M-3, C-M-3 all for this purpose, again for all digits 0-9. The thing is, I simply never use them in this way, and this is a huge waste of really useful keyboard shortcuts you could be using for other things.

To reclaim them, add the following to your emacs config file, with credit to this reddit post:

;; unset C- and M- digit keys
(dotimes (n 10)
  (global-unset-key (kbd (format "C-%d" n)))
  (global-unset-key (kbd (format "M-%d" n)))

You can then still access the digit prefixes with C-M-3 and similar but you now have all the C- and M- digit combinations to play with.

One great way to make use of this is to set up your own key map using one of these newly freed-up combinations. I set C-1 as the prefix key for my own map:

;; set up my own map
(define-prefix-command 'bjm-map)
(global-set-key (kbd "C-1") 'bjm-map)

Then I assign some keys to that map, for example

(define-key bjm-map (kbd "m") 'mu4e)
(define-key bjm-map (kbd "g") 'bjm/open-gcal-agenda)

Now I can use C-1 m to open mu4e for my email, and C-1 g to view my google calendar.

-1:-- Use your digits and a personal key map for super shortcuts (Post Ben Maughan)--L0--C0--January 07, 2017 08:18 PM

emacshorrors: No Obvious Way To Do It Properly

You can often catch me griping way more about an “Emacs API” than Emacs Lisp, often without any explanation how the interfaces Emacs provides could be possibly worse than the language used for this. Here comes a not too obvious example of the issue at hand that will not go away if you were to rewrite the code in question in <insert favorite toy language>:

(defvar flyspell-prog-text-faces
  '(font-lock-string-face font-lock-comment-face font-lock-doc-face)
  "Faces corresponding to text in programming-mode buffers.")

(defun flyspell-generic-progmode-verify ()
  "Used for `flyspell-generic-check-word-predicate' in programming modes."
  ;; (point) is next char after the word. Must check one char before.
  (let ((f (get-text-property (- (point) 1) 'face)))
    (memq f flyspell-prog-text-faces)))

The above is helper code for Flyspell to detect whether point is in a string or comment. While this is most certainly the easiest way to achieve this goal, it isn’t what the manual recommends. What you’re supposed to do instead is to use the syntax table parser by accessing its state via syntax-ppss and checking slots 3 and 4[1]. The issue with that approach is that it’s not that simple, see the sources of Smartparens for the ugly details, more specifically the sp-point-in-string-or-comment function.

Let’s take a step back. Why exactly do we need to check the syntactical state of Emacs? To understand what’s going on, you’ll need to know that the first step when building a major mode for a programming language is to set up the syntax tables so that Emacs can recognize strings and comments properly. Once this has been done, both syntax highlighting and movement work correctly. This hack has been done back then to have that special case dealt with in a performant manner, albeit it only works properly for languages looking like C or Lisp. Single characters can be marked as having a special syntax, with extra slots for two-character sequences. In Lisp, a semicolon would start the comment and every newline would terminate one, in C the comment starters and enders would be /* and */. If you’re using a language with more complicated rules than that, there is the crutch of writing a custom syntax-propertize-function that walks over an arbitrary buffer region and applies syntax properties manually. This is how triple-quoted strings are implemented in python.el, the same technique is used to highlight a S-expression comment in scheme.el which starts with #; and extends to the following S-expression.

Unfortunately, not all modes play well. Some resort to just painting over the thing to be a comment or string by applying the right face to it and sacrifice proper movement over symbols and S-expressions. It shouldn’t be surprising that given the choice to fix faulty modes or circumvent a cumbersome API, plenty Emacs hackers pick the latter and get more robust handling for free. After all, practicality beats purity, right?

[1]I’ve seen at least one mode checking slot 8 instead. A detailed analysis of the difference is left to the reader as exercise.
-1:-- No Obvious Way To Do It Properly (Post Vasilij Schneidermann)--L0--C0--January 05, 2017 07:28 PM

emacspeak: Fun With TTS (Voxin) And Ladspa

Fun With TTS (Voxin) And Ladspa

1 Executive Summary

Voxin 1.6 — AKA ViaVoice Outloud — no longer requires that the
Emacspeak TTS server be built as a 32-bit binary. This means that
installing Voxin on 64-bit systems is now significantly easier since
you no longer need to install 32-bit versions of TCL, TCLX, and the
dependencies needed by library In addition to
easing the installation process, not needing 32-bit binaries means
that the Emacspeak Outloud server can now take advantage of audio
processing such as that provided by LADSPA.

2 Going 64-Bit: Upgrading To Voxin 1.6

  1. Install Voxin-1.6 or later from Voxin.
  2. Update Emacspeak from GitHub (this will be part of the next
    public release).
  3. Rebuild the binary in the servers/linux-outloud
cd servers/linux-outloud && make clean && make

If all goes well, you'll now have a 64-bit version of
You can now run the Outloud server as servers/outloud.
In about a year's time, servers/32-outloud will move to
servers/obsolete, as will the associated servers/32-speech-server
and servers/ssh-32-outloud.

3 Applying LADSPA Effects Processing To TTS

With a 64-bit build of in place, we can now call on
installed LADSPA plugins to apply digital sound processing to TTS
output. To experiment with the possibilities, see some of the
virtual sound devices defined in servers/linux-outloud/asoundrc.
Copy over that file to your ~/.asoundrc after updating it to match
your sound setup — you'll likely need to change the default
sound-card to match your setup.
You can now set environment variable ALSA_DEFAULT to one of the
tts_<effect> virtual devices — and have the Outloud server apply
the specified LADSPA effect to the generated TTS. Here is an example:

cd servers 
(export ALSA_DEFAULT=tts_reverb; ./outloud)

4 The Best Is Yet To Come …

The possibilities are endless — ALSA with LADSPA provides a rich
suite of audio processing possibilities.

5 Acknowledgements

I'd like to acknowledge Gilles Casse for his work over the years on
ensuring that Linux users have access to good quality TTS. Outloud
would have been dead a long time ago if it weren't for his continued
efforts toward keeping the lights on. His newest creation, libvoxin
that forms the crux of Voxin-1.6 is an excellent piece of engineering
that is likely to help Outloud survive for the future on modern Linux
distros. Note that Gilles is also the primary author of the Emacspeak
ESpeak server.

-1:-- Fun With TTS (Voxin) And Ladspa (Post T. V. Raman ( 04, 2017 07:13 PM

John Stevenson: camcorder.el - Creating Animated Gifs & Videos for Spacemacs

Using animated gifs are a lightweight way to show Emacs in action, as can be seen at Emacs Gifs.

I am creating a workshop on developing Clojure with Spacemacs, so here is a little guide as to how I create animated gifs and videos for this workshop directly from Emacs itself using camcorder.el.

There are several different ways to create animated gifs and so far I have found camcorder.el to be the easiest. This approach has been tested on Ubuntu Linux 16.10.

Why an animated gif rather than video?

Being an image format, animated gifs can be used like any other image. So they are easy to include in websites. They will also play continually without any additional code.

Animated gifs are typically smaller in size, so are quicker to download than video and use less resources too.

Some small amount of quality is lost when converting to an animated gif. However, using the optimised and hence slower conversion gives a reasonable quality. I am still experimenting with the setting though to see if I can make the conversion better.


camcorder.el enables you to create a screen capture specifically of Emacs. When run, a new Emacs frame is created with the contents of the current buffer and your actions in that frame are recorded as a video.

Then camcorder.el can convert the video to an animated gif.

Requirements for camcorder.el

The camcorder.el package itself does not actually do the recording or even the converting, its simply a convienient way to manage other tools without having to leave Emacs.

For a quick and simple conversion from video to animated gif you can select ffmpeg. If you want to optimise the size of the resulting animated gif then select the combination of mplayer and imagemagick.

sudo apt-get install recordmydesktop mplayer imagemagick

Add camcorder to Spacemacs

As far as I am aware there is not yet a Spacemacs layer that includes camcorder.el. So instead we add camcorder.el as an additional package.

Edit your ~/.spacemacs configuration file and find dotspacemacs-additional-packages. Then add camcorder to that list of packages:

dotspacemacs-additional-packages &apos(camcorder)

You will need to either reload your Spacemacs configuration with SPC f e R or restart Emacs.

Capturing with camcorder

Run either camcorder-record or camcorder-mode to record Emacs. You are first prompted to give the file name of the video output (ogv video format).

Once you have specified the video name, a 3 second countdown will run before starting the recording.

Command Description
M-x camcorder-record Open a new Emacs frame and record
M-x camcorder-mode Record current Emacs frame
F11 Toggle pause pause
F12 Stop recording and save video

Screencasts are generated in ogv format, although if you hack camcorder.el you could change the video format used and even the video capture tool (recordmydesktop).

As capturing creates a video file, you can edit that file with tools such as OpenShot or Blender video editors. So if you make a small error or want to shorten (or lengthen) a part of the video, then edit it before you convert it to gif.

Converting with camcorder

You can convert the videos you generated during capturing, or any other supported video type. So you can also used camcorder.el if you recorded Emacs (or other tools) separately.

Run the M-x camcorder-convert-to-gif and you are prompted for the video file to convert to an animated gif.

Spacemacs - camcorder - convert-to-gif

Tweaking camcorder.el

I only made one tweak to camcorder.el, to change the size of the frame created for the capture.

I changed the height from 20 to 32 and the width from 65 to 120. These sizes provided more space to see the Spacemacs menu as I demonstrate features. When creating screen captures I run my desktop at a resolution of 1360x768 and a Spacemacs font size of 16 (Ubuntu Mono).

(defcustom frame-parameters
&apos((name . "camcorder.el Recording - F12 to Stop - F11 to Pause/Resume")
(height . 32)
(width . 120)
(top . 80))
"Parameters used on the recording frame.
See `make-frame&apos."
:type &apos(alist :key-type symbol :value-type sexp))

In Summary

The package camcorder.el provides a simple 2-step process to create animated gif images of Emacs demos. You can tweak the script easily and can also use different tools to do the screen capture.

Animated gifs are very easy to distribute, especially on web pages and github pages sites. With this process you also have a video version of the demo too.

Keeping your demos short, between 10 and 20 seconds, typically makes the animated gifs easy to follow. So think about what the most important point you are trying to convey when you are creating a new animated gif.

-1:-- camcorder.el - Creating Animated Gifs &amp; Videos for Spacemacs (Post)--L0--C0--January 02, 2017 08:59 PM

John Stevenson: Kanban in Emacs Org-Mode to Get More Work Done

A Kanban board is a way to visualise your work and help you get more work done. You organise your work into tasks that need completeing and use the board to show the state of each card. Kanban encourages you to get work finished before starting new work.

The amazing Emacs Org-mode can be used to create a very fast and easy to use Kanban board that is with you where ever you are.

Update: Using Org-mode doesnt give me everything I want from a Kanban board, but it was an interesting exersice. For now, I am just sticking to my list view of a Kanban board.

Org-mode is built into Emacs / Spacemacs so there is no need to install any packages or layers for any of the following.

Designing a kanban board

The columns on your kanban board represent the state of work and represent your typical workflow. You can represent the states as the most generic todo, doing, done workflow, or anything more specific that adds value to how you manage work.

I have been using kanban for a while, so I am using a five stage workflow: planning, in progress, blocked, review, done

  • planning - work I’d like to do that needs organising so I can do it.
  • in progress - what I am currently working on. I try and keep this to a minimum so I get things done
  • blocked - things I’ve started working on but currently arent able to complete
  • review - work I have completed. Check if there are any follow on tasks or lessons learnt
  • done - things I have completed. Gives feeling of satisfaction

Creating Org-mode stages

Its easy to create your own Org-mode stages, to represent the state of work in your Kanban board. Please see my earlier article on Configuring Emacs Org-Mode to Managing Your Tasks

Create an Org-mode file

Create a new file by opening a new buffer M-x find-files and type in the new file name, ending in .org. Any files with a .org filename extension will automatically set the Emacs major mode to Org-mode.

I use a file called for my kanban board.

Spacemacs Emacs
SPC f f C-x C-f
M-x spacemacs/helm-find-files M-x find-files

Create a kanban board

Lets assume you created a file called Edit this file and create a table for the kanban board. You can start creating ths manually by typing | for the table layout or use M-x org-table-create and enter the number of columns and rows for the table. For example, to create for a table with 5 columns and 3 rows, you would speciify 5x3

Add the names of the kanban board in the first row of the table. If you did not use M-x org-table-create then add a header row with M-x org-table-insert-hline.

In my kanban board, this gives

Emacs Org-mode table as Kanban board

Adding tasks to the kanban board

Each item on the board represents a task and we use the Org-mode interal link to jump from the board to the details of the task. To create a link of the same name as the task, simply type the name inside double square brakets [[]].

Emacs Org-mode table as Kanban board - task entry

Moving the tasks across the board

Its easy enough to move the columns around with Alt - <arrow-keys> in org-mode, but there is not a single keybinding to move a cell.

To move the individual tasks between the columns use selective cut and paste:

  • Move the cursor to the cell you want to move and use C-c C-x C-w
  • Use TAB to move to the new cell
  • Paste/Yank the value into the new cell using C-c C-x C-y

However, simply moving the task does not update the Org-mode stage. As each task is a link, I can click on that link and I am taken to the task and can easily update the task stage to match the board.

It would be great if moving the tasks on the board updated the associated task stage and vice versa.

El Kanban - updating the board from task stage changes

I found the El Kanban package that will updated the kanban board based on the task org-mode stages. This uses the Org-mode table format directive that you run each time you want to update the board.

I installed this package and it did pull in my custom org-mode stages for the headers. Unfortunately it did not pull in the tasks to the board, so I will either need to fix the package or find another solution.

Any suggestions are more than welcome.


Thank you.

-1:-- Kanban in Emacs Org-Mode to Get More Work Done (Post)--L0--C0--January 01, 2017 10:14 PM

Bryan Murdock: Adventures in Arch Linux

I just installed Arch Linux on my 4th machine. It has been fun and painful. Painfully fun? I have learned a lot and that is always fun. There.

I have loved using Ubuntu over the last several (eight, I think) years. Ubuntu is easy to install and most things Just Work. It's easy to find packages for most software I want to run, and there is lots of help on the internet for accomplishing whatever you want to accomplish with Ubuntu. My frustrations have been that even though you can find instructions for getting whatever software you want to run, it's not always a simple apt-get install. Sometimes it's configuring a PPA source and sometimes it's compiling from source. Sometimes a PPA works and serves you well for years and then suddenly it disappears. Another frustration is out of date packages, and full system upgrades in general. Keeping up with the latest emacs was a chore. Going from one release of Ubuntu to another works surprisingly well, but it's a big enough chore that I keep putting it off. One of my desktop machines at home was still running 12.04 up until yesterday. That's 5 and a half years old now!

These concerns led me to Arch. It seems to be addressing them beautifully. Every package I have wanted is either in the main repositories and I can install it with a simple pacman command, or it's in the AUR and I can install it with a simple yaourt command. There are no releases of Arch, they just continually update the packages that are in the repositories. Staying up to date is always the same pacman command to upgrade your installed packages. There are times where you have to take some manual steps to fix an interaction between two packages, or to switch from a package that has been made obsolete by another newer package, but that's fairly rare, well documented, and you just deal with it a little at a time when the situations come up. With Ubuntu dist-upgrades you had to deal with many of those scenarios all at once, every 6 months if you were keeping fully up to date. With Arch, keeping up with the latest emacs happens without me even realizing it.

Where Arch is not as nice as Ubuntu is in the installation. With Arch it's all manual. What you should do is pretty well documented, but you have to type all the commands yourself and make decisions about alternative ways of doing various things. It's a fun learning experience as I mentioned at the beginning of this post, but not a process that I really enjoyed repeating over and over. This is really where Ubuntu made it's name. The nice package system and repositories were straight from Debian originally, but the installer and default configurations are what made Ubuntu so accessible. There was a joke in the early Ubuntu days that Ubuntu was an African word meaning, "can't install Debian."

It turns out that there's a distribution with a name meaning, "can't install Arch." It's Antergos. It really is just a graphical installer on top of Arch. Once it's done you are running Arch with some Antergos chosen configuration, which is exactly what I wanted. It does feel like it's still early days for this project. On one laptop I tried Antergos on it didn't have the wifi drivers I needed. I had to go back to plain arch and figure out how to load the driver by hand in order to complete the installation (that should be a blog post of its own). On another machine once the Antergos install was done the display manager would crash with a webkitwebprocess coredump. The antergos forums told me how to switch to lxdm and that fixed my problem (probably should be another blog post). I don't think a linux beginner would have enjoyed that process, but overall Antergos looks promising. Mostly I'm looking forward to never needing to do a fresh install on any of those machines ever again.

-1:-- Adventures in Arch Linux (Post Bryan ( 31, 2016 05:23 PM

Flickr tag 'emacs': 2016-12-27a How do I capture ideas #mobile #workflow #emacs #org #drawing #krita

sachac posted a photo:

2016-12-27a How do I capture ideas #mobile #workflow #emacs #org #drawing #krita

-1:-- 2016-12-27a How do I capture ideas #mobile #workflow #emacs #org #drawing #krita (Post sachac ( 29, 2016 04:45 AM

Chris Wellons: Domain-Specific Language Compilation in Elfeed

Last night I pushed another performance enhancement for Elfeed, this time reducing the time spent parsing feeds. It’s accomplished by compiling, during macro expansion, a jQuery-like domain-specific language within Elfeed.

Heuristic parsing

Given the nature of the domain — an under-specified standard and a lack of robust adherence — feed parsing is much more heuristic than strict. Sure, everyone’s feed XML is strictly conforming since virtually no feed reader tolerates invalid XML (thank you, XML libraries), but, for the schema, the situation resembles the de facto looseness of HTML. Sometimes important or required information is missing, or is only available in a different namespace. Sometimes, especially in the case of timestamps, it’s in the wrong format, or encoded incorrectly, or ambiguous. It’s real world data.

To get a particular piece of information, Elfeed looks in a number of different places within the feed, starting with the preferred source and stopping when the information is found. For example, to find the date of an Atom entry, Elfeed first searches for elements in this order:

  1. <published>
  2. <updated>
  3. <date>
  4. <modified>
  5. <issued>

Failing to find any of these elements, or if no parsable date is found, it settles on the current time. Only the updated element is required, but published usually has the desired information, so it goes first. The last three are only valid for another namespace, but are useful fallbacks.

Before Elfeed even starts this search, the XML text is parsed into an s-expression using xml-parse-region — a pure Elisp XML parser included in Emacs. The search is made over the resulting s-expression.

For example, here’s a sample from the Atom specification.

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="">

  <title>Example Feed</title>
  <link href=""/>
    <name>John Doe</name>

    <title>Atom-Powered Robots Run Amok</title>
    <link rel="alternate" href=""/>
    <summary>Some text.</summary>


Which is parsed to into this s-expression.

((feed ((xmlns . ""))
       (title () "Example Feed")
       (link ((href . "")))
       (updated () "2003-12-13T18:30:02Z")
       (author () (name () "John Doe"))
       (id () "urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6")
       (entry ()
              (title () "Atom-Powered Robots Run Amok")
              (link ((rel . "alternate")
                     (href . "")))
              (id () "urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a")
              (updated () "2003-12-13T18:30:02Z")
              (summary () "Some text."))))

Each XML element is converted to a list. The first item is a symbol that is the element’s name. The second item is an alist of attributes — cons pairs of symbols and strings. And the rest are its children, both string nodes and other elements. I’ve trimmed the extraneous string nodes from the sample s-expression.

A subtle detail is that xml-parse-region doesn’t just return the root element. It returns a list of elements, which always happens to be a single element list, which is the root element. I don’t know why this is, but I’ve built everything to assume this structure as input.

For Elfeed, all namespaces stripped from both elements and attributes to make parsing simpler. As I said, it’s heuristic rather than strict, so namespaces are treated as noise.

A domain-specific language

Coding up Elfeed’s s-expression searches in straight Emacs Lisp would be tedious, error-prone, and difficult to understand. It’s a lot of loops, assoc, etc. So instead I invented a jQuery-like, CSS selector-like, domain-specific language (DSL) to express these searches concisely and clearly.

For example, all of the entry links are “selected” using this expression:

(feed entry link [rel "alternate"] :href)

Reading right-to-left, this matches every href attribute under every link element with the rel="alternate" attribute, under every entry element, under the feed root element. Symbols match element names, two-element vectors match elements with a particular attribute pair, and keywords (which must come last) narrow the selection to a specific attribute value.

Imagine hand-writing the code to navigate all these conditions for each piece of information that Elfeed requires. The RSS parser makes up to 16 such queries, and the Atom parser makes as many as 24. That would add up to a lot of tedious code.

The package (included with Elfeed) that executes this query is called “xml-query.” It comes in two flavors: xml-query and xml-query-all. The former returns just the first match, and the latter returns all matches. The naming parallels the querySelector() and querySelectorAll() DOM methods in JavaScript.

(let ((xml (elfeed-xml-parse-region)))
  (xml-query-all '(feed entry link [rel "alternate"] :href) xml))

;; => ("")

That date search I mentioned before looks roughly like this. The * matches text nodes within the selected element. It must come last just like the keyword matcher.

(or (xml-query '(feed entry published *))
    (xml-query '(feed entry updated *))
    (xml-query '(feed entry date *))
    (xml-query '(feed entry modified *))
    (xml-query '(feed entry issued *))

Over the past three years, Elfeed has gained more and more of these selectors as it collects more and more information from feeds. Most recently, Elfeed collects author and category information provided by feeds. Each new query slows feed parsing a little bit, and it’s a perfect example of a program slowing down as it gains more features and capabilities.

But I don’t want Elfeed to slow down. I want it to get faster!

Optimizing the domain-specific language

Just like the primary jQuery function ($), both xml-query and xml-query-all are functions. The xml-query engine processes the selector from scratch on each invocation. It examines the first element, dispatches on its type/value to apply it to the input, and then recurses on the rest of selector with the narrowed input, stopping when it hits the end of the list. That’s the way it’s worked from the start.

However, every selector argument in Elfeed is a static, quoted list. Unlike user-supplied filters, I know exactly what I want to execute ahead of time. It would be much better if the engine didn’t have to waste time reparsing the DSL for each query.

This is the classic split between interpreters and compilers. An interpreter reads input and immediately executes it, doing what the input tells it to do. A compiler reads input and, rather than execute it, produces output, usually in a simpler language, that, when evaluated, has the same effect as executing the input.

Rather than interpret the selector, it would be better to compile it into Elisp code, compile that into byte-code, and then have the Emacs byte-code virtual machine (VM) execute the query each time it’s needed. The extra work of parsing the DSL is performed ahead of time, the dispatch is entirely static, and the selector ultimately executes on a much faster engine (byte-code VM). This should be a lot faster!

So I wrote a function that accepts a selector expression and emits Elisp source that implements that selector: a compiler for my DSL. Having a readily-available syntax tree is one of the big advantages of homoiconicity, and this sort of function makes perfect sense in a lisp. For the external interface, this compiler function is called by a new pair of macros, xml-query* and xml-query-all*. These macros consume a static selector and expand into the compiled Elisp form of the selector.

To demonstrate, remember that link query from before? Here’s the macro version of that selection, but only returning the first match. Notice the selector is no longer quoted. This is because it’s consumed by the macro, not evaluated.

(xml-query* (feed entry title [rel "alternate"] :href) xml)

This will expand into the following code.

(catch 'done
  (dolist (v xml)
    (when (and (consp v) (eq (car v) 'feed))
      (dolist (v (cddr v))
        (when (and (consp v) (eq (car v) 'entry))
          (dolist (v (cddr v))
            (when (and (consp v) (eq (car v) 'title))
              (let ((value (cdr (assq 'rel (cadr v)))))
                (when (equal value "alternate")
                  (let ((v (cdr (assq 'href (cadr v)))))
                    (when v
                      (throw 'done v))))))))))))

As soon as it finds a match, it’s thrown to the top level and returned. Without the DSL, the expansion is essentially what would have to be written by hand. This is exactly the sort of leverage you should be getting from a compiler. It compiles to around 130 byte-code instructions.

The xml-query-all* form is nearly the same, but instead of a throw, it pushes the result into the return list. Only the prologue (the outermost part) and the epilogue (the innermost part) are different.

Parsing feeds is a hot spot for Elfeed, so I wanted the compiler’s output to be as efficient as possible. I had three goals for this:

  • No extraneous code. It’s easy for the compiler to emit unnecessary code. The byte-code compiler might be able to eliminate some of it, but I don’t want to rely on that. Except for the identifiers, it should basically look like a human wrote it.

  • Avoid function calls. I don’t want to pay function call overhead, and, with some care, it’s easy to avoid. In the xml-query* expansion, the only function call is throw, which is unavoidable. The xml-query-all* version makes no function calls whatsoever. Notice that I used assq rather than assoc. First, it only needs to match symbols, so it should be faster. Second, assq has its own byte-code instruction (158) and assoc does not.

  • No unnecessary memory allocations. The xml-query* expansion makes no allocations. The xml-query-all* version only conses once per output, which is the minimum possible.

The end result is at least as optimal as hand-written code, but without the chance of human error (typos, fat fingering) and sourced from an easy-to-read DSL.


In my tests, the xml-query macros are a full order of magnitude faster than the functions. Yes, ten times faster! It’s an even bigger gain than I expected.

In the full picture, xml-query is only one part of parsing a feed. Measuring the time starting from raw XML text (as delivered by cURL) to a list of database entry objects, I’m seeing an overall 25% speedup with the macros. The remaining time is dominated by xml-parse-region, which is mostly out of my control.

With xml-query so computationally cheap, I don’t need to worry about using it more often. Compared to parsing XML text, it’s virtually free.

When it came time to validate my DSL compiler, I was really happy that Elfeed had a test suite. I essentially rewrote a core component from scratch, and passing all of the unit tests was a strong sign that it was correct. Many times that test suite has provided confidence in changes made both by me and by others.

I’ll end by describing another possible application: Apply this technique to regular expressions, such that static strings containing regular expressions are compiled into Elisp/byte-code via macro expansion. I wonder if situationally this would be faster than Emacs’ own regular expression engine.

-1:-- Domain-Specific Language Compilation in Elfeed (Post)--L0--C0--December 27, 2016 09:46 PM

Chris Wellons: Some Performance Advantages of Lexical Scope

I recently had a discussion with Xah Lee about lexical scope in Emacs Lisp. The topic was why lexical-binding exists at a file-level when there was already lexical-let (from cl-lib), prompted by my previous article on JIT byte-code compilation. The specific context is Emacs Lisp, but these concepts apply to language design in general.

Until Emacs 24.1 (June 2012), Elisp only had dynamically scoped variables — a feature, mostly by accident, common to old lisp dialects. While dynamic scope has some selective uses, it’s widely regarded as a mistake for local variables, and virtually no other languages have adopted it.

Way back in 1993, Dave Gillespie’s deviously clever lexical-let macro was committed to the cl package, providing a rudimentary form of opt-in lexical scope. The macro walks its body replacing local variable names with guaranteed-unique gensym names: the exact same technique used in macros to create “hygienic” bindings that aren’t visible to the macro body. It essentially “fakes” lexical scope within Elisp’s dynamic scope by preventing variable name collisions.

For example, here’s one of the consequences of dynamic scope.

(defun inner ()
  (setq v :inner))

(defun outer ()
  (let ((v :outer))

;; => :inner

The “local” variable v in outer is visible to its callee, inner, which can access and manipulate it. The meaning of the free variable v in inner depends entirely on the run-time call stack. It might be a global variable, or it might be a local variable for a caller, direct or indirect.

Using lexical-let deconflicts these names, giving the effect of lexical scope.

(defvar v)

(defun lexical-outer ()
  (lexical-let ((v :outer))

;; => :outer

But there’s more to lexical scope than this. Closures only make sense in the context of lexical scope, and the most useful feature of lexical-let is that lambda expressions evaluate to closures. The macro implements this using a technique called closure conversion. Additional parameters are added to the original lambda function, one for each lexical variable (and not just each closed-over variable), and the whole thing is wrapped in another lambda function that invokes the original lambda function with the additional parameters filled with the closed-over variables — yes, the variables (e.g. symbols) themselves, not just their values, (e.g. pass-by-reference). The last point means different closures can properly close over the same variables, and they can bind new values.

To roughly illustrate how this works, the first lambda expression below, which closes over the lexical variables x and y, would be converted into the latter by lexical-let. The #: is Elisp’s syntax for uninterned variables. So #:x is a symbol x, but not the symbol x (see print-gensym).

;; Before conversion:
(lambda ()
  (+ x y))

;; After conversion:
(lambda (&rest args)
  (apply (lambda (x y)
           (+ (symbol-value x)
              (symbol-value y)))
         '#:x '#:y args))

I’ve said on multiple occasions that lexical-binding: t has significant advantages, both in performance and static analysis, and so it should be used for all future Elisp code. The only reason it’s not the default is because it breaks some old (badly written) code. However, lexical-let doesn’t realize any of these advantages! In fact, it has worse performance than straightforward dynamic scope with let.

  1. New symbol objects are allocated and initialized (make-symbol) on each run-time evaluation, one per lexical variable.

  2. Since it’s just faking it, lexical-let still uses dynamic bindings, which are more expensive than lexical bindings. It varies depending on the C compiler that built Emacs, but dynamic variable accesses (opcode varref) take around 30% longer than lexical variable accesses (opcode stack-ref). Assignment is far worse, where dynamic variable assignment (varset) takes 650% longer than lexical variable assignment (stack-set). How I measured all this is a topic for another article.

  3. The “lexical” variables are accessed using symbol-value, a full function call, so they’re even slower than normal dynamic variables.

  4. Because converted lambda expressions are constructed dynamically at run-time within the body of lexical-let, the resulting closure is only partially byte-compiled even if the code as a whole has been byte-compiled. In contrast, lexical-binding: t closures are fully compiled. How this works is worth its own article.

  5. Converted lambda expressions include the additional internal function invocation, making them slower.

While lexical-let is clever, and occasionally useful prior to Emacs 24, it may come at a hefty performance cost if evaluated frequently. There’s no reason to use it anymore.

Constraints on code generation

Another reason to be weary of dynamic scope is that it puts needless constraints on the compiler, preventing a number of important optimization opportunities. For example, consider the following function, bar:

(defun bar ()
  (let ((x 1)
        (y 2))
    (+ x y)))

Byte-compile this function under dynamic scope (lexical-binding: nil) and disassemble it to see what it looks like.

(byte-compile #'bar)
(disassemble #'bar)

That pops up a buffer with the disassembly listing:

0       constant  1
1       constant  2
2       varbind   y
3       varbind   x
4       constant  foo
5       call      0
6       discard
7       varref    x
8       varref    y
9       plus
10      unbind    2
11      return

It’s 12 instructions, 5 of which deal with dynamic bindings. The byte-compiler doesn’t always produce optimal byte-code, but this just so happens to be nearly optimal byte-code. The discard (a very fast instruction) isn’t necessary, but otherwise no more compiler smarts can improve on this. Since the variables x and y are visible to foo, they must be bound before the call and loaded after the call. While generally this function will return 3, the compiler cannot assume so since it ultimately depends on the behavior foo. Its hands are tied.

Compare this to the lexical scope version (lexical-binding: t):

0       constant  1
1       constant  2
2       constant  foo
3       call      0
4       discard
5       stack-ref 1
6       stack-ref 1
7       plus
8       return

It’s only 8 instructions, none of which are expensive dynamic variable instructions. And this isn’t even close to the optimal byte-code. In fact, as of Emacs 25.1 the byte-compiler often doesn’t produce the optimal byte-code for lexical scope code and still needs some work. Despite not firing on all cylinders, lexical scope still manages to beat dynamic scope in performance benchmarks.

Here’s the optimal byte-code, should the byte-compiler become smarter someday:

0       constant  foo
1       call      0
2       constant  3
3       return

It’s down to 4 instructions due to computing the math operation at compile time. Emacs’ byte-compiler only has rudimentary constant folding, so it doesn’t notice that x and y are constants and misses this optimization. I speculate this is due to its roots compiling under dynamic scope. Since x and y are no longer exposed to foo, the compiler has the opportunity to optimize them out of existence. I haven’t measured it, but I would expect this to be significantly faster than the dynamic scope version of this function.

Optional dynamic scope

You might be thinking, “What if I really do want x and y to be dynamically bound for foo?” This is often useful. Many of Emacs’ own functions are designed to have certain variables dynamically bound around them. For example, the print family of functions use the global variable standard-output to determine where to send output by default.

(let ((standard-output (current-buffer)))
  (princ "value = ")
  (prin1 value))

Have no fear: With lexical-binding: t you can have your cake and eat it too. Variables declared with defvar, defconst, or defvaralias are marked as “special” with an internal bit flag (declared_special in C). When the compiler detects one of these variables (special-variable-p), it uses a classical dynamic binding.

Declaring both x and y as special restores the original semantics, reverting bar back to its old byte-code definition (next time it’s compiled, that is). But it would be poor form to mark x or y as special: You’d de-optimize all code (compiled after the declaration) anywhere in Emacs that uses these names. As a package author, only do this with the namespace-prefixed variables that belong to you.

The only way to unmark a special variable is with the undocumented function internal-make-var-non-special. I expected makunbound to do this, but as of Emacs 25.1 it does not. This could possibly be considered a bug.

Accidental closures

I’ve said there are are absolutely no advantages to lexical-binding: nil. It’s only the default for the sake of backwards-compatibility. However, there is one case where lexical-binding: t introduces a subtle issue that would otherwise not exist. Take this code for example (and nevermind prin1-to-string for a moment):

;; -*- lexical-binding: t; -*-

(defun function-as-string ()
    (prin1 (lambda () :example) (current-buffer))

This creates and serializes a closure, which is one of Elisp’s unique features. It doesn’t close over any variables, so it should be pretty simple. However, this function will only work correctly under lexical-binding: t when byte-compiled.

;; => "(closure ((temp-buffer . #<buffer  *temp*>) t) nil :example)"

The interpreter doesn’t analyze the closure, so just closes over everything. This includes the hidden variable temp-buffer created by the with-temp-buffer macro, resulting in an abstraction leak. Buffers aren’t readable, so this will signal an error if an attempt is made to read this function back into an s-expression. The byte-compiler fixes this by noticing temp-buffer isn’t actually closed over and so doesn’t include it in the closure, making it work correctly.

Under lexical-binding: nil it works correctly either way:

;; -> "(lambda nil :example)"

This may seem contrived — it’s certainly unlikely — but it has come up in practice. Still, it’s no reason to avoid lexical-binding: t.

Use lexical scope in all new code

As I’ve said again and again, always use lexical-binding: t. Use dynamic variables judiciously. And lexical-let is no replacement. It has virtually none of the benefits, performs worse, and it only applies to let, not any of the other places bindings are created: function parameters, dotimes, dolist, and condition-case.

-1:-- Some Performance Advantages of Lexical Scope (Post)--L0--C0--December 22, 2016 02:33 AM

Marcin Borkowski: Generating links to info buffers

When writing an email about Emacs, e.g. on the help-gnu-emacs mailing list, I often want to include a link to an Info node. It is easy in Org-mode, where you can use org-store-link (however, by default, the link does not export “properly” to HTML, though, i.e., you don’t get the link to the online Emacs manual) and then org-insert-link. But even if you don’t use Org-mode, there is a nice trick:
-1:-- Generating links to info buffers (Post)--L0--C0--December 18, 2016 06:55 PM

Mark Hershberger: How I chased down a PHP mode indention bug in Emacs

(When I posted this to reddit, someone pointed out that I could have gotten the same information from c-show-syntactic-information by hitting C-c C-s. Awesome!)

I was getting irritated by emacs refusing to indent PHP code properly when I was working on a MediaWiki extension.  I’ve run into this before, but, today, in order to procrastinate a little, I ran into it again with the following code:

try {
    $object->method( ARG );

It kept trying to put $object the same level as try, so it would end up with:

try {
$object->method( ARG );

So I chased it down. I used C-h k to find the function being called. After a bit of edebug, I found that the code in the function c-indent-line being called was essentially:

(c-get-syntactic-indentation (c-guess-basic-syntax))

In fact, doing M-; (c-get-syntactic-indentation (c-guess-basic-syntax)) RET when point was sitting on $ gave the result 4 when it tried to indent and 0 when it didn’t.

(Ok, the code had two more levels of indention than the above, so it was giving 12 and 8, but let’s not get carried away with details.)

Now, running M-x php-mode RET (i.e. none of the added configuration in .dir-locals.el) gave the proper indention. In my .dir-locals.el, though, I had set up a c-offsets-alist that mostly worked with bits and pieces copied from all over.

Running just M-; (c-guess-basic-syntax) RET returned ((statement-block-intro 1379)) so I figured I needed to add (statement-block-intro . +) to my c-offsets-list.

I added that, and it worked. And now I know how to chase down indention bugs.

(Header image by Ivo Kruusamägi [CC BY-SA 4.0 (], via Wikimedia Commons.)

-1:-- How I chased down a PHP mode indention bug in Emacs (Post hexmode)--L0--C0--December 17, 2016 06:21 PM

Timo Geusch: Converting files from DOS to UNIX file formats using Emacs

What do you do if you don’t have dos2unix, need to convert a file or three from DOS (or Mac) format to UNIX format, but all you have is Emacs? Well, of course you use Emacs for the file conversion, Read More

The post Converting files from DOS to UNIX file formats using Emacs appeared first on The Lone C++ Coder's Blog.

-1:-- Converting files from DOS to UNIX file formats using Emacs (Post Timo Geusch)--L0--C0--December 14, 2016 02:43 AM

Jorgen Schäfer: Race conditions in Emacs' process filter functions

Race conditions are always fun. They are particularly surprising when they happen in a single-threaded program like Emacs. I just ran into this problem for the second time.

When Emacs reads data from a process, it executes a filter functionassociated with that process. The idea is that this filter function handles the data, and after it returns, Emacs continues with other work. Due to some peculiarities of Emacs, it is easily possible for filter functions to run while other filter functions are running, seemingly randomly.

Emacs runs filter functions whenever it reads data from a process, even if this happens while a filter function is already running. And there are many places where Emacs reads data from processes. Even if you knew all the functions that do so, you do not control or know about the code Emacs runs most of the time. For example, if you create a buffer in a particular mode, that mode’s hook is run, which could execute pretty much anything.

I first encountered this problem in Circe, my IRC client for Emacs. In a filter function, it can open a new buffer in a specific mode (a query). That mode can have flyspell active, Emacs spell checker. But flyspell starts a new process for spell checking. Starting a new process can cause Emacs to read from processes, running filter functions. This resulted in a difficult to debug problem that happened only when a user received two messages in very close succession.

You can read my comments on the problem in the Circe source code.

I just ran into the same problem again in Elpy, my Python development environment. To my surprise, Emacs’ process-send-string does not only send data, it also reads data from the process and handles it in the same call. In this particular case, the usual sequence of action is to run an initialization function, and then set a flag that initialization happened, so no need to run it again. But the initialization function sends data to a process. And the filter function for this process might call the original function. Which would notice that no initialization happened, send data, which would read data, which would notice that no initialization happened yet, so send data again … stack overflow.

Solution? A dynamic scope “lock” (not a real lock as it does not block, but close enough).

(defvar my-lock nil)
(when (not my-lock)
(let ((my-lock t))

For a single-threaded application like Emacs, this looks completely useless. But it works.

-1:-- Race conditions in Emacs' process filter functions (Post Jorgen Schäfer ( 13, 2016 04:20 PM

Grant Rettke: Why Do Keyboards Matter?


Wada (creator of the Happy Hacking Keyboard): because keyboards are accessories to PC makers, they focus on minimizing the manufacturing costs. But that’s incorrect. It’s in HHKB’s slogan, but when America’s cowboys were in the middle of a trip and their horse died, they would leave the horse there. But even if they were in the middle of a desert, they would take their saddle with them. The horse was a consumable good, but the saddle was an interface that their bodies had gotten used to. In the same vein, PCs are consumable goods, while keyboards are important interfaces.

The HHKB shirt says: 「馬の鞍」- saddle of a horse.

-1:-- Why Do Keyboards Matter? (Post Grant)--L0--C0--December 09, 2016 09:18 PM

Alex Schroeder: Cider Woes

I’m trying to get Cider to work on my laptop running OSX and failing. This makes me unhappy because I managed to do it on a Windows machine with lots of Cygwin so I felt that getting it to work on a Mac should be a breeze.

Basically, what I have set up my little project using lein and can run it from the command line and from eshell, and I use Clojure Mode, but when I run M-x cider-jack-in I see is this in my *Messages* buffer:

Starting nREPL server via /Users/alex/bin/lein update-in :dependencies conj \[org.clojure/tools.nrepl\ \"0.2.12\"\ \:exclusions\ \[org.clojure/clojure\]\] -- update-in :plugins conj \[cider/cider-nrepl\ \"0.15.0-SNAPSHOT\"\] -- repl :headless...
Mark set
nREPL server started on 53929
[nREPL] Establishing direct connection to localhost:53929 ...
error in process filter: nrepl--direct-connect: [nREPL] Direct connection failed
error in process filter: [nREPL] Direct connection failed

But now I have three buffers named *nrepl-server ... with contents such as nREPL server started on port 53872 on host - nrepl://

This reminds me… My Mac keeps deleting my /etc/hosts file which means that localhost is borked:

$ ping localhost
ping: cannot resolve localhost: Unknown host

So I added it back to /etc/hosts. localhost

Then I killed the three nrepl-server buffers and tried C-c M-j again.

And it works. Phew!

Sometimes writing these posts just helps me focus as I’m troubleshooting.


-1:-- Cider Woes (Post)--L0--C0--December 08, 2016 08:25 PM

Chen Bin (redguardtoo): Use Perforce inside Emacs

Perforce is a proprietary VCS.

1 Perforce workflow

  • p4 set to set up
  • p4 login to login
  • p4 client creates a new work space to tell server which files to check out or to ignore
  • p4 sync ... to download files
  • Files are read-only by default. You need p4 edit file to make files writable before editing
  • p4 add files to add new files. p4 revert to revert edited file to it original status and lose the local changes
  • p4 change to create a pending change. Then p4 submit -c #changelist to actually submit code to main branch. Pending change gives you a chance to tweak the change before submit
  • Or p4 submit -d"description" file to submit a single file directly

2 My solution

Perforce cygwin portable is not recommended.

I suggest using p4's windows native executable on both Cygwin and Windows.

Perforce server has one unique URL for every physical file on the disk. If I only feed p4 that URL, the operation is always successful.

BTW, I'm confident that my solution is better.

2.1 Emacs

I provides p4edit, p4revert, and p4submit to edit/revert/submit current file,

;; {{ perforce utilities
(defvar p4-file-to-url '("" "")
  "(car p4-file-to-url) is the original file prefix
(cadr p4-file-to-url) is the url prefix")

(defun p4-generate-cmd (opts)
  (format "p4 %s %s"
          (replace-regexp-in-string (car p4-file-to-url)
                                    (cadr p4-file-to-url)
(defun p4edit ()
  "p4 edit current file."
  (shell-command (p4-generate-cmd "edit"))
  (read-only-mode -1))

(defun p4submit (&optional file-opened)
  "p4 submit current file.
If FILE-OPENED, current file is still opened."
  (interactive "P")
  (let* ((msg (read-string "Say (ENTER to abort):"))
         (open-opts (if file-opened "-f leaveunchanged+reopen -r" ""))
         (full-opts (format "submit -d '%s' %s" msg open-opts)))
    ;; (message "(p4-generate-cmd full-opts)=%s" (p4-generate-cmd full-opts))
    (if (string= "" msg)
        (message "Abort submit.")
      (shell-command (p4-generate-cmd full-opts))
      (unless file-opened (read-only-mode 1))
      (message (format "%s submitted."
                       (file-name-nondirectory buffer-file-name))))))

(defun p4revert ()
  "p4 revert current file."
  (shell-command (p4-generate-cmd "revert"))
  (read-only-mode 1))
;; }}

(defun prog-mode-hook-setup ()
  (when (string-match-p "DIR/PROJ1/"
                        (if buffer-file-name buffer-file-name ""))
    (setq-local p4-file-to-url '("^.*DIR/PROJ1"
(add-hook 'prog-mode-hook prog-mode-hook-setup)

As a bonus tip, if you use find-file-in-project, insert below code into prog-mode-hook to view any perforce change inside Emacs,

(setq-local ffip-diff-backends
            '((ivy-read "p4 change to show:"
                        (split-string (shell-command-to-string "p4 changes //depot/development/DIR/PROJ1/...")
                        :action (lambda (i)
                                  (if (string-match "^ Change \\([0-9]*\\)" i)
                                      (shell-command-to-string (format "p4 describe -du -db %s"
                                                                       (match-string 1 i))))))
              "p4 diff -du -db //depot/development/DIR/PROJ1/..."))

2.2 Bash Shell

Other operations are finished in Bash Shell,

# {{ Perforce, I hope I will never use it
if [ "$OS_NAME" = "CYGWIN" ]; then
    function p4() {
        export PWD=`cygpath -wa .`
        /cygdrive/c/Program\ Files/Perforce/p4.exe $@

# p4 workflow:
#   # basic setup
#   p4 set P4CLIENT=clientname  # set your default client
#   p4 set P4PORT=SERVER:1666
#   p4 set P4USER=username
#   p4 client # create/edit client, client views selected files
#   # checkout code
#   p4 sync [-f] //depot/project-name/path/...
#   p4 edit file[s]
#   ... do some editing ...
#   # submit code
#   either `p4 submit -d"say hi" file` or `p4 change`
#   I recommend `p4 change` because you can edit files list before submit happens.
#   After `p4 change`,  `p4 submit -c changelist#` to actually submit change.
alias p4clr='p4 diff -sr | p4 -x - revert' # like `git reset HEAD`
alias p4blame='p4 annotate -c -db ' # could add -a see deleted lines
alias p4cr='p4 submit -f leaveunchanged+reopen -r'
alias reviewcl='ccollab addchangelist new'
alias p4pending='p4 changes -s pending' # add ... for current directory
alias p4untrack='find . -type f| p4 -x - fstat >/dev/null'
alias p4v='p4 resolve' # after `p4 sync ...`, maybe resolve
alias p4r='p4 revert' # discard changes
alias p4e='p4 edit'
alias p4s='p4 submit'
alias p4sr='p4 submit -f submitunchanged+reopen' #submit&reopen
alias p4up='p4 sync ...' # synchronize from current directory
alias p4o='p4 opened' # list opened files
alias p4c='p4 changes' # create a new pending change
alias p4chg='p4 change' # create a pending change
alias p4d='p4 diff -du -db'
alias p4ds='p4 diff -du -db | lsdiff' # diff summary, patchutils required
alias p4i='p4 integrate'
alias p4unsh='p4 unshelve -s' # Usage: p4unsh changelist#, like `git stash apply`
alias p4h='p4 changes -m 1 ...' # show the head change

function p4mypending {
    local P4USERNAME="`p4 user -o | grep '^User:\s' | sed 's/User:\s\([a-bA-B0-9]*\)/\1/g'`"
    p4 changes -s pending -u $P4USERNAME

function p4shelved {
    local P4USERNAME="`p4 user -o | grep '^User:\s' | sed 's/User:\s\([a-bA-B0-9]*\)/\1/g'`"
    p4 changes -s shelved -u $P4USERNAME # add ... for current directory

function p4cmp {
    if [ -z "$1" ]; then
        echo "Usage: p4cmp changelist-number changelist-number"
        p4 diff2 -dub -q -u ...@$1 ...@$2

function p4dl {
    # git diff
    p4 diff -du -db $@ | vim -c "set syntax=diff" -R -
function p4sh(){
    # show specific change or the latest change
    if [ -z "$1" ]; then
        p4 changes | python ~/bin/ | awk '{print $2}' | xargs -i p4 describe -du {} | vim -c "set syntax=diff" -R -
        p4 describe -du -db $@ | vim -c "set syntax=diff" -R -

function p4lp {
    #like `git log -p`
    p4 changes $@ | awk '{print $2}' | xargs -i p4 describe -du {} | less -F

function p4mlp {
    #like `git log -p`
    p4 changes -u $P4USERNAME $@ | awk '{print $2}' | xargs -i p4 describe -du {} | less -F

function p4adddir(){
    if [ -z "$1" ]; then
        echo "Usage: p4adddir directory"
        find $1 -type f -print | p4 -x - add

# p4's suggestion,
# @google "assing variable from bash to perl in a bash script"
function p4l(){
    # p4 log
    if [ -z "$1" ]; then
        # show the full log
        p4 changes -l ... | less
        # p4log since-changelist-number
        p4 changes -l ...@$1,#head|perl -pe "if(\$_=~/$1/){ last;};"

function p4ml(){
    # my p4 log
    if [ -z "$1" ]; then
        # show the full log
        p4 changes -l -u $P4USERNAME ... | less
        # p4log since-changelist-number
        p4 changes -l -u $P4USERNAME ...@$1,#head|perl -pe "if(\$_=~/$1/){ last;};"
# }}

-1:-- Use Perforce inside Emacs (Post Chen Bin)--L0--C0--December 05, 2016 11:51 AM

Timo Geusch: What happened to XEmacs?

I used XEmacs quite a lot in the 2000s before I switched back to the more stable GNU Emacs. That was back then before GNU Emacs offered a stable official Windows build when XEmacs did, and at the time I Read More

The post What happened to XEmacs? appeared first on The Lone C++ Coder's Blog.

-1:-- What happened to XEmacs? (Post Timo Geusch)--L0--C0--December 03, 2016 06:40 PM

Alex Schroeder: Emacs Avent of Code Day 2

First question of day 2: You start on key 5 on a keypad as shown below. With instructions to go up, down, left or right. Ignore instructions that would take you off the keypad.

1 2 3
4 5 6
7 8 9

The example input provided for the four numbers is shown below. At the end of each line you’ll end up on the key which is part of the answer. In this example the correct answer is 1985.



(let ((pos 5)
      (code nil)
      (instructions '((U L L)
		      (R R D D D)
		      (L U R D L)
		      (U U U U D))))
  (dolist (instruction instructions)
    (dolist (move instruction)
      (case move
	((U) (when (> pos 3) (setq pos (- pos 3))))
	((D) (when (< pos 7) (setq pos (+ pos 3))))
	((L) (when (not (= (mod pos 3) 1)) (setq pos (- pos 1))))
	((R) (when (> (mod pos 3) 0) (setq pos (+ pos 1))))))
    (push pos code))
  (reverse code))

For the second star, I was too lazy to figure out how to compute the next position and decided to implement tables instead:

  2 3 4
5 6 7 8 9
  A B C

Given a direction, I just list all the possible positions and the results. It’s crude, but it works.

(let ((pos 5)
      (code nil)
      (instructions '((U L L)
		      (R R D D D)
		      (L U R D L)
		      (U U U U D))))
(dolist (instruction instructions)
    (dolist (move instruction)
      (setq pos (funcall move pos)))
    (push pos code))
  (reverse code))

(defun U (pos)
  (case pos
    ((3) 1)
    ((6) 2)
    ((7) 3)
    ((8) 4)
    ((A) 6)
    ((B) 7)
    ((C) 8)
    ((D) 'B)
    (t pos)))

(defun D (pos)
  (case pos
    ((1) 3)
    ((2) 6)
    ((3) 7)
    ((4) 8)
    ((6) 'A)
    ((7) 'B)
    ((8) 'C)
    ((B) 'D)
    (t pos)))

(defun L (pos)
  (case pos
    ((6) 5)
    ((3) 2)
    ((7) 6)
    ((B) 'A)
    ((4) 3)
    ((8) 7)
    ((C) 'B)
    ((9) 8)
    (t pos)))

(defun R (pos)
  (case pos
    ((5) 6)
    ((2) 3)
    ((6) 7)
    ((A) 'B)
    ((3) 4)
    ((7) 8)
    ((B) 'C)
    ((8) 9)
    (t pos)))


-1:-- Emacs Avent of Code Day 2 (Post)--L0--C0--December 03, 2016 12:13 AM

Manuel Uberti: Format XML in Emacs

Since I deal with many SOAP web services for work, I also have to deal with lots of XML files. In the application server log they usually appear on one line, which makes the reading unnecessarily painful. Fortunately, Emacs coupled with the handy xmllint comes to the rescue.

First, I installed xmllint with: sudo apt-get install libxml2-utils

Now I could format my XML buffers with: C-x h C-u M-| xmllint -format - RET

However, this is Emacs, so there is always a more productive way to solve problems. Why should I have to input the options for xmllint or remember the necessary key bindings all the times?

(defun mu-xml-format ()
  "Format an XML buffer with `xmllint'."
  (shell-command-on-region (point-min) (point-max)
                           "xmllint -format -"
                           (current-buffer) t
                           "*Xmllint Error Buffer*" t))

Since I only need this utility in XML buffers, I bound it for nxml-mode-map.

(use-package nxml-mode                  ; XML editing
  :mode "\\.xml\\'"
  :bind (:map nxml-mode-map
              ("C-c m f" . mu-xml-format))

-1:-- Format XML in Emacs (Post)--L0--C0--December 03, 2016 12:00 AM

William Denton: Emacs in Conforguration

On top of Ruby I now have Emacs itself set up in Conforguration. On a new bare machine I can git clone two GitHub repositories, run some scripts, and after a lot of compiling my entire Emacs setup will be ready and waiting. (Note: works on Ubuntu, and probably other Debian-based systems but I don’t know for sure.)

sudo apt-get install git
git clone
mkdir -p src
cd src
git clone
cd conforguration/dotfiles
source ~/.bashrc
cd ../scripts

Then running emacs will download and install a couple of dozen packages. When that’s done you’ll see exactly what my Emacs is like. If you don’t want to go that far, skip the .emacs.d repository download, but make sure your PATH is set up to point to Emacs in the right place under /usr/local/src/.

Once Emacs is installed you can run the Conforguration scripts inside it, but it’s nice they also work outside, for those scary early minutes on a new machine when Emacs is not installed.

-1:-- Emacs in Conforguration (Post William Denton)--L0--C0--December 02, 2016 03:09 AM

emacspeak: Follow-Up: Soundscapes On The Emacspeak Audio Desktop

Follow-Up: Soundscapes On The Emacspeak Audio Desktop

1 Executive Summary

Nearly a year ago, I blogged here about Soundscapes on the emacspeak
Audio Desktop. That article ended with this following paragraph:

I implemented package soundscape to create a platform that would let me experiment
with different tools that aid in concentration. After using Soundscapes for about a week,
I have also found that it reduces some of the fatigue that results from having to listen
to synthetic text-to-speech for extended periods. The true value (if any) of this package
will be a function of how heavily I find myself using it six months from now --- as a
metric, complete success might mean that in mid-2016, I still have automatic soundscapes
turned on.

2 And Nearly A Year Later …

I have not found the need to turn off Soundscapes in Emacspeak. As
conjectured, it has definitely increased my productivity, specifically
in terms of staying focused on a given task at hand. Over the year,
I've also augmented the emacspeak Audio Desktop with support for
binaural audio — see module sox-gen — which provides a collection of
binaural themes for use during different times of the day. Binaural
themes generated by that module overlay Emacspeak Soundscapes to
provide an ideal auditory environment for use over headphones.

3 Soundscape Enhancements

Since the publication of the original article, Emacspeak Soundscapes
have been enhanced with additional sounds from Emacspeak Soundscapes have been updated to take
advantage of Boodler's limited abilities in the areas of spatial
positioning. I typically use Soundscapes with one of several virtual
ALSA devices that have been configured to apply different Ladspa
effects such as reverb or crossfeed depending on the ambient
environment where I am working — this significantly improves the
spatialization of soundscapes being played — see file
ladspa-asoundrc. Finally, the mapping of Soundscapes to various Emacs
modes has also been tuned. — see table below.

Soundscape (Mood) List Of Major Modes
BirdSongs shell term
BlopEchoes elfeed-search
Bonfire calendar diary
BuddhaLoop comint
Cavern prog
ChangingLoops special
ChangingLoopsPitches lisp-interaction
Drip message gnus-summary gnus-article gnus-group mspools vm-presentation vm mail twittering jabber-roster jabber-chat erc
LoopStew emacspeak-m-player
NoStormYet fundamental
RainForever Info help Man Custom messages-buffer
RainSounds magit vc
Still text view
SurfWaves w3 eww
TonkSpace tabulated-list
WaterFlow dired

4 Summary

As outlined in a previous article, sound on Linux provides unending
possibilities with respect to innovation, here's looking forward to
better things to come.

-1:-- Follow-Up: Soundscapes On The Emacspeak Audio Desktop (Post T. V. Raman ( 29, 2016 11:06 PM

emacshorrors: When Data Becomes Code

If you’ve hung out on #emacs for a while, chances are you’ve been recommended ibuffer for advanced buffer management (like, killing many buffers at once). There is a horror lurking beneath its pretty interface though and it starts out with a customizable:

(defcustom ibuffer-always-compile-formats (featurep 'bytecomp)
  "If non-nil, then use the byte-compiler to optimize `ibuffer-formats'.
This will increase the redisplay speed, at the cost of loading the
elisp byte-compiler."
  :type 'boolean
  :group 'ibuffer)


(defun ibuffer-compile-make-eliding-form (strvar elide from-end-p)
  (let ((ellipsis (propertize ibuffer-eliding-string 'font-lock-face 'bold)))
    (if (or elide (with-no-warnings ibuffer-elide-long-columns))
        `(if (> strlen 5)
             ,(if from-end-p
                  ;; FIXME: this should probably also be using
                  ;; `truncate-string-to-width' (Bug#24972)
                  `(concat ,ellipsis
                           (substring ,strvar
                                      (string-width ibuffer-eliding-string)))
                   ,strvar (- strlen (string-width ,ellipsis)) nil ?.)

(defun ibuffer-compile-make-substring-form (strvar maxvar from-end-p)
  (if from-end-p
      ;; FIXME: not sure if this case is correct (Bug#24972)
      `(truncate-string-to-width str strlen (- strlen ,maxvar) nil ?\s)
    `(truncate-string-to-width ,strvar ,maxvar nil ?\s)))

(defun ibuffer-compile-make-format-form (strvar widthform alignment)
  (let* ((left `(make-string tmp2 ?\s))
         (right `(make-string (- tmp1 tmp2) ?\s)))
       (setq tmp1 ,widthform
             tmp2 (/ tmp1 2))
       ,(pcase alignment
          (:right `(concat ,left ,right ,strvar))
          (:center `(concat ,left ,strvar ,right))
          (:left `(concat ,strvar ,left ,right))
          (_ (error "Invalid alignment %s" alignment))))))

(defun ibuffer-compile-format (format)
  (let ((result nil)
        ;; We use these variables to keep track of which variables
        ;; inside the generated function we need to bind, since
        ;; binding variables in Emacs takes time.
        (vars-used ()))
    (dolist (form format)
       ;; Generate a form based on a particular format entry, like
       ;; " ", mark, or (mode 16 16 :right).
       (if (stringp form)
           ;; It's a string; all we need to do is insert it.
           `(insert ,form)
         (let* ((form (ibuffer-expand-format-entry form))
                (sym (nth 0 form))
                (min (nth 1 form))
                (max (nth 2 form))
                (align (nth 3 form))
                (elide (nth 4 form)))
           (let* ((from-end-p (when (cl-minusp min)
                                (setq min (- min))
                  (letbindings nil)
                  (outforms nil)
                  min-used max-used strlen-used)
             (when (or (not (integerp min)) (>= min 0))
               ;; This is a complex case; they want it limited to a
               ;; minimum size.
               (setq min-used t)
               (setq strlen-used t)
               (setq vars-used '(str strlen tmp1 tmp2))
               ;; Generate code to limit the string to a minimum size.
               (setq minform `(progn
                                (setq str
                                        `(- ,(if (integerp min)
             (when (or (not (integerp max)) (> max 0))
               (setq max-used t)
               (cl-pushnew 'str vars-used)
               ;; Generate code to limit the string to a maximum size.
               (setq maxform `(progn
                                (setq str
                                        (if (integerp max)
                                (setq strlen (string-width str))
                                (setq str
                                        'str elide from-end-p)))))
             ;; Now, put these forms together with the rest of the code.
             (let ((callform
                    ;; Is this an "inline" column?  This means we have
                    ;; to get the code from the
                    ;; `ibuffer-inline-columns' alist and insert it
                    ;; into our generated code.  Otherwise, we just
                    ;; generate a call to the column function.
                    (ibuffer-aif (assq sym ibuffer-inline-columns)
                        (nth 1 it)
                      `(,sym buffer mark)))
                   ;; You're not expected to understand this.  Hell, I
                   ;; don't even understand it, and I wrote it five
                   ;; minutes ago.
                    (if (get sym 'ibuffer-column-summarizer)
                        ;; I really, really wish Emacs Lisp had closures.
                        ;; FIXME: Elisp does have them now.
                        (lambda (arg sym)
                            (let ((ret ,arg))
                              (put ',sym 'ibuffer-column-summary
                                   (cons ret (get ',sym
                      (lambda (arg _sym)
                        `(insert ,arg))))
                   (mincompform `(< strlen ,(if (integerp min)
                   (maxcompform `(> strlen ,(if (integerp max)
               (if (or min-used max-used)
                   ;; The complex case, where we have to limit the
                   ;; form to a maximum or minimum size.
                     (when (and min-used (not (integerp min)))
                       (push `(min ,min) letbindings))
                     (when (and max-used (not (integerp max)))
                       (push `(max ,max) letbindings))
                      (if (and min-used max-used)
                          `(if ,mincompform
                             (if ,maxcompform
                        (if min-used
                            `(when ,mincompform
                          `(when ,maxcompform
                     (push `(setq str ,callform
                                  ,@(when strlen-used
                                      `(strlen (string-width str))))
                     (setq outforms
                           (append outforms
                                   (list (funcall insertgenfn 'str sym)))))
                 ;; The simple case; just insert the string.
                 (push (funcall insertgenfn callform sym) outforms))
               ;; Finally, return a `let' form which binds the
               ;; variables in `letbindings', and contains all the
               ;; code in `outforms'.
               `(let ,letbindings
    ;; We don't want to unconditionally load the byte-compiler.
    (funcall (if (or ibuffer-always-compile-formats
                     (featurep 'bytecomp))
             ;; Here, we actually create a lambda form which
             ;; inserts all the generated forms for each entry
             ;; in the format string.
             `(lambda (buffer mark)
                (let ,vars-used
                  ,@(nreverse result))))))

(defun ibuffer-recompile-formats ()
  "Recompile `ibuffer-formats'."
  (setq ibuffer-compiled-formats
        (mapcar #'ibuffer-compile-format ibuffer-formats))
  (when (boundp 'ibuffer-filter-format-alist)
    (setq ibuffer-compiled-filter-formats
          (mapcar (lambda (entry)
                    (cons (car entry)
                          (mapcar (lambda (formats)
                                    (mapcar #'ibuffer-compile-format formats))
                                  (cdr entry))))

(defun ibuffer-clear-summary-columns (format)
  (dolist (form format)
    (when (and (consp form)
               (get (car form) 'ibuffer-column-summarizer))
      (put (car form) 'ibuffer-column-summary nil))))

(defun ibuffer-check-formats ()
  (when (null ibuffer-formats)
    (error "No formats!"))
  (let ((ext-loaded (featurep 'ibuf-ext)))
    (when (or (null ibuffer-compiled-formats)
              (null ibuffer-cached-formats)
              (not (eq ibuffer-cached-formats ibuffer-formats))
              (null ibuffer-cached-eliding-string)
              (not (equal ibuffer-cached-eliding-string ibuffer-eliding-string))
              (eql 0 ibuffer-cached-elide-long-columns)
              (not (eql ibuffer-cached-elide-long-columns
                        (with-no-warnings ibuffer-elide-long-columns)))
              (and ext-loaded
                   (not (eq ibuffer-cached-filter-formats
                   (and ibuffer-filter-format-alist
                        (null ibuffer-compiled-filter-formats))))
      (message "Formats have changed, recompiling...")
      (setq ibuffer-cached-formats ibuffer-formats
            ibuffer-cached-eliding-string ibuffer-eliding-string
            ibuffer-cached-elide-long-columns (with-no-warnings ibuffer-elide-long-columns))
      (when ext-loaded
        (setq ibuffer-cached-filter-formats ibuffer-filter-format-alist))
      (message "Formats have changed, recompiling...done"))))

Another weird one is that the extracted autoloads for ibuffer-ext.el reside in ibuffer.el, but that’s the lesser evil of the two.

Credits go to holomorph for discovering that maintenance nightmare.

-1:-- When Data Becomes Code (Post Vasilij Schneidermann)--L0--C0--November 28, 2016 09:31 AM

Sebastian Wiesner: Bye Bye Emojis: Emacs Hates MacOS

Two years ago Emacs 24.4 dramatically improved support for MacOS (which was still OS X back then) with a new Core Text based font rendering engine. Among better and more consistent rendering this change—with the right fontset configuration—enabled colored emoji. Not the most important feature, but nice to have and doing no harm either.

In Emacs 25.1 it is gone. What I initially thought was a mistake in my configuration turned out to be a NEWS entry (emphasis mine):

On the OS X Cocoa (“Nextstep”) port, multicolor font (such as color emoji) display is disabled. […] This will be enabled again once it is also implemented in Emacs on free operating systems. […]

Let’s sink this in: The Emacs developers deliberately disabled a feature that was working perfectly fine for MacOS users just because it is not available for free systems1. What a daft decision.

In the FSF’s little ivory tower of free software happiness this decision surely makes sense, but as a user, as a maintainer of a popular Emacs package that tries to deliver a great user interface, I feel like I’m being given the finger. It is a clear message that no matter what we contribute2 as MacOS users we will always be second-class citizen in Emacs land3.

We are not welcome, and never will be.

  1. As if it was our fault that Linux is notoriously lagging behind on desktop.

  2. I wrote all of Flycheck on a Macbook.

  3. Even though the Emacs maintainer is a MacOS user himself, and wanted better support for MacOS when he took up that position. He explicitly disagreed with that change, though.

-1:-- Bye Bye Emojis: Emacs Hates MacOS (Post Sebastian Wiesner)--L0--C0--November 21, 2016 11:00 PM