Jorgen Schäfer: Circe 2.1 released

We just released version 2.1 of Circe, a Client for IRC in Emacs.

The package is available from github, MELPA stable and MELPA unstable. The latter will track further development changes, so use at your own risk.


  • New option: circe-inhibit-nick-highlight-function – this allows you to disable nick highlighting in some messages.
  • New extension: circe-new-day-notifier.el – show date changes in chat buffers. (Thanks to Pásztor János!)
  • Improve Bitlbee support by adding a default port (6667) and disabling lagmon if it is used.
  • Improved buttonizing of various references, like PEP links or Emacs debbugs references.
  • Fix a bug that would confuse Emacs with lots of nil faces
  • Lots of other bug fixes.

Thanks to Chunyang Xu, Pásztor János, Riccardo Murri and Vasilij Schneidermann for their contributions!

-1:-- Circe 2.1 released (Post Jorgen Schäfer ( 29, 2015 02:27 PM

Marcin Borkowski: Fixing mml-attach-file using advice

I use mu4e as my MUA, and I’m quite happy with it. (I toyed with the idea of setting up Gnus, but I can’t afford quitting my job to find the time for that;-).) However, one thing with the message mode (which mu4e uses) did annoy me a lot: the attachments.
-1:-- Fixing mml-attach-file using advice (Post)--L0--C0--November 28, 2015 10:56 PM

Pragmatic Emacs: Get pop-up help for keybindings with which-key

In my post on showing unfinished keybindings early, commenter hmelman points out which-key which is a package that give you pop-up help for partially completed keystrokes. Simply install the package from MELPA and add


to your emacs config file. Now type a prefix key like C-x and wait one second (this delay can be customised) and a new window will open with a summary of the possible completions for that prefix. There are some nice screenshots at the project web page so look at those to get the idea.

This is a great way to remind yourself of commands or discover new ones. For example try C-x 8 and wait to see what pops up – I’m sure you’ll find some commands under that prefix that you didn’t know about!

-1:-- Get pop-up help for keybindings with which-key (Post Ben Maughan)--L0--C0--November 27, 2015 08:30 PM

Chen Bin (redguardtoo): Code completion for HTML/JS/CSS in Emacs

Now company-mode is very popular in Emacs user. It supports code completion out of box.

Its company-etags.el can use TAGS created by ctags/etags.

Setup is easy.

Step 1, create TAGS:

cd ~/myproj/script; ctags -e '*.js'; cd ~/myproj/style; ctags -e '*.css'

Step 2, add below code into "~/.emacs.d/init.el":

(setq tags-table-list (list "~/myproj/script/TAGS" "~/myproj/style/TAGS"))

That's OK for most programming languages.

But not enough for web developers who use web-mode.

First, you need upgrade web-mode to latest version so that when inputting "btn-" or "data-" code completion still works.

Second, you need add below code into your "~/.emacs.d/init.el",

(eval-after-load 'company-etags
     (add-to-list 'company-etags-modes 'web-mode)))

You need above setup because web-mode conditionally inherit from prog-mode. You can `M-x eval-expression (derived-mode-p 'prog-mode)` to test whether current mode inherits from prog-mode. Thanks for Dmitry Gutov pointing out that js2-mode actually inherits from prog-mode because it inherits from js-mode which inheris from prog-mode.

Tested on Emacs23.4 and Emacs24.4.

Screenshot of my setup:


-1:-- Code completion for HTML/JS/CSS in Emacs (Post Chen Bin)--L0--C0--November 27, 2015 11:43 AM

sachachua: Org Mode tables and fill-in quizzes – Latin verb conjugation drills in Emacs

I was looking for a Latin verb conjugation drill similar to these ones for and nouns and pronouns. I liked the instant feedback and the ability to quickly get hints. I couldn’t find an online drill I liked, though, so I made my own with Emacs and Org. (Because… why not?)

I wrote some code that would take a table like this:

present – 1st sing. – ago / agere agO
present – 2nd sing. – ago / agere agis
present – 3rd sing. – ago / agere agit
present – 1st plu. – ago / agere agimus
present – 2nd plu. – ago / agere agitis
present – 3rd plu. – ago / agere agunt
imperfect – 1st sing. – ago / agere agEbam
imperfect – 2nd sing. – ago / agere agEbAs
imperfect – 3rd sing. – ago / agere agEbat
imperfect – 1st plu. – ago / agere agEbAmus
imperfect – 2nd plu. – ago / agere agEbAtis
imperfect – 3rd plu. – ago / agere agEbant
future – 1st sing. – ago / agere agam
future – 2nd sing. – ago / agere agEs
future – 3rd sing. – ago / agere agEt
future – 1st plu. – ago / agere agEmus
future – 2nd plu. – ago / agere agEtis
future – 3rd plu. – ago / agere agent

I can call my/make-fill-in-quiz to get a quiz buffer that looks like this. If I get stuck, ? shows me a hint in the echo area.


To make it easier, I’ve left case-fold-search set to nil so that I don’t have to match the case (uppercase vowels = macrons), but I can set case-fold-search to t if I want to make sure I’ve got the macrons in the right places.

Here’s the code to display the quiz buffer.

     (require 'widget)
     (defun my/check-widget-value (widget &rest ignore)
       "Provide visual feedback for WIDGET."
        ((string= (widget-value widget) "?")
         ;; Asking for hint
         (message "%s" (widget-get widget :correct))
         (widget-value-set widget ""))
        ;; Use string-match to obey case-fold-search 
          (concat "^"
                  (regexp-quote (widget-get widget :correct))
          (widget-value widget))
         (message "Correct")
         (goto-char (widget-field-start widget))
         (goto-char (line-end-position))
         (insert "✓")
         (widget-forward 1)

   (defun my/make-fill-in-quiz (&optional quiz-table)
     "Create an fill-in quiz for the Org table at point.
The Org table's first column should have the questions and the second column 
should have the answers."
     (interactive (list (org-babel-read-table)))
     (with-current-buffer (get-buffer-create "*Quiz*")
       (let ((inhibit-read-only t))
       (mapc (lambda (row)
               (widget-insert (car row))
               (widget-insert "\t")
               (widget-create 'editable-field
                              :size 15
                              :correct (cadr row)
                              :notify 'my/check-widget-value)
               (widget-insert "\n"))    
       (widget-create 'push-button
                      :table quiz-table
                      :notify (lambda (widget &rest ignore)
                                (my/make-fill-in-quiz (widget-get widget :table))) 
       (use-local-map widget-keymap)
       (goto-char (point-min))
       (widget-forward 1)
       (switch-to-buffer (current-buffer))))

Incidentally, I generated the table above from a larger table of Latin verb conjugations in the appendix of Wheelock’s Latin, specified like this:

#+NAME: present-indicative-active
| laudO    | moneO   | agO    | audiO   | capiO   |
| laudAs   | monEs   | agis   | audIs   | capis   |
| laudat   | monet   | agit   | audit   | capit   |
| laudAmus | monEmus | agimus | audImus | capimus |
| laudAtis | monEtis | agitis | audItis | capitis |
| laudant  | monent  | agunt  | audiunt | capiunt |

#+NAME: imperfect-indicative-active
| laudAbam   | monEbam   | agEbam   | audiEbam   | capiEbam   |
| laudAbas   | monEbas   | agEbAs   | audiEbAs   | capiEbas   |
| laudAbat   | monEbat   | agEbat   | audiEbat   | capiEbat   |
| laudAbAmus | monEbAmus | agEbAmus | audiEbAmus | capiEbAmus |
| laudAbAtis | monEbAtis | agEbAtis | audiEbAtis | capiEbAtis |
| laudAbant  | monEbant  | agEbant  | audiEbant  | capiEbant  |

#+NAME: future-indicative-active
| laudAbO    | monEbO    | agam   | audiam     | capiam     |
| laudAbis   | monEbis   | agEs   | audiEs     | capiEs     |
| laudAbit   | monEbit   | agEt   | audiet     | capiet     |
| laudAbimus | monEbimus | agEmus | audiEmus   | capiEmus   |
| laudAbitis | monEbitis | agEtis | audiEtis   | capiEtis   |
| laudAbunt  | monEbunt  | agent  | audient    | capient    |

with the code:

#+begin_src emacs-lisp :var present=present-indicative-active :var imperfect=imperfect-indicative-active :var future=future-indicative-active
  (defun my/label-latin-with-verbs (table verbs persons tense)
    (apply 'append
           (-zip-with (lambda (row person) 
                        (-zip-with (lambda (word verb)
                                     (list word (format "%s - %s - %s" tense person verb)))
                                   row verbs))
                      table (-cycle persons))))
  (apply 'append 
         (mapcar (lambda (tense)
                    (symbol-value tense)
                    '("laudo / laudare" "moneo / monEre" "ago / agere" "audiO / audIre" "capiO / capere")
                    '("1st sing." "2nd sing." "3rd sing." "1st plu." "2nd plu." "3rd plu.")
                    (symbol-name tense)))
                 '(present imperfect future)))


This uses dash.el for the -zip-with and -cycle functions. There’s probably a much better way to process the lists, but I’m still getting the hang of thinking properly functionally… =)

Anyway, I’m sure it will be handy for a number of other quiz-like things. org-drill and org-drill-table will probably come in handy for flashcards, too!

The post Org Mode tables and fill-in quizzes – Latin verb conjugation drills in Emacs appeared first on sacha chua :: living an awesome life.

-1:-- Org Mode tables and fill-in quizzes – Latin verb conjugation drills in Emacs (Post Sacha Chua)--L0--C0--November 27, 2015 02:22 AM

Raimon Grau: Managing dockers (and some emacs help)

Docker containers are a great way to isolate an application. You don't need to pollute your host machine with all the dependencies, and makes it easy to deploy it everywhere.

I'm using docker on my local box for some apps I can't (or don't want) to install, one of them being openresty (until there's a guix package (or I manage to create it), I'll keep it that way).  But using docker for local development is a bit more complex than just running an app there.

First, a few tips for using docker for local development:

Using docker without sudo

By default, docker asks for superuser permissions in order to comunicate to docker server. You have to create a user group called docker and add your user to that group. After that, you can run docker commands as your regular user.

Building docker images and running them

In the beginning,I was creating lots of images, because every time you change any code in your app, the image has to be rebuilt. To build an image, being in your app directory (where your Dockerfile is and your code is), you normally do 'docker build .'  Then, after building, the last line of the output contains the hash for that image in a line like: 'Successfully built 01b4cd5d3e'.  you normally want to run it afterwards, so you'll 'docker run -t -i 01b4cd5d3e bash', to get in a console inside it.  I just wrote a couple of shell functions to ease the loop of build and run.

function dbuild {
export DBUILD=$(docker build . | grep "Successfully built" | col -b | awk '{print $3}')
echo -n $DBUILD

function dattach () {
docker run -t -i $DBUILD bash

With this I can save the copypasting of the image id.

There's yet a better trick to not have to rebuild the image every time, which is to mount a directory of the host machine inside the container.  For this to happen, we have to add the '-v' flat to docker run. Now, in the docker file, we'll create a directory for our code to be in, and just mount it the current directory in that place. In my case, the directory is '/opt/apisonator'.
function dattach ()  {
docker run -v $(pwd):/opt/apisonator/:rw -t -i $DBUILD bash

With this, we have a pretty slick way to run our code and tests inside docker, with full access to the docker, but also being in synch with the outside.

Sooo.... what about emacs?

In case you have many images and containers, you'll probably want to do bulk actions on them (running them, deleting, etc.). Emacs has 2 modes for interacting with docker: docker.el and docker-tramp.el .

The first one (docker.el) uses tabulated-list to show all docker images and containers in your machine, and allows you to run, pause,kill, push or pull images and containers.

Docker-tramp.el allows you to use tramp to access the files inside the container from your host emacs. That's pretty pretty neat:  c-x c-f //docker:012f23e2fb/var/logs/.... .

Again, emacs saves the day.

-1:-- Managing dockers (and some emacs help) (Post Raimon Grau ( 27, 2015 02:13 AM

Pragmatic Emacs: Show unfinished keybindings early

When you start typing a keybinding that has several parts to it (like C-x C-s to save), if you pause after entering the first part (called the prefix key, C-x in this example), then emacs shows the bit you have typed so far in the echo area (the line at the bottom of the screen where messages appear).

By default, emacs waits 1 second before echoing the keystroke, but I like to see it sooner. Of course this can be customised – add the following to your emacs config file to set it to 0.1 seconds:

;; Show unfinished keystrokes early.
(setq echo-keystrokes 0.1)


Commenter hmelman points out which-key which is a package that give you pop up help for partially completed keystrokes. For many, this probably supersedes the echoing of help above, and is worthy of its own post.

-1:-- Show unfinished keybindings early (Post Ben Maughan)--L0--C0--November 26, 2015 09:28 PM

Irreal: Emacs for the CEO

Back when Josh Stella was coding, he lived in Emacs. Like many of us, he performed most of his everyday tasks<img src=—" class="wp-smiley" style="height: 1em; max-height: 1em;" />mail, calendar, documents, coding<img src=—" class="wp-smiley" style="height: 1em; max-height: 1em;" />from within Emacs. Decades later, he'd become the CEO/co-founder of Luminal and had left Emacs behind.

Like many developers<img src=—" class="wp-smiley" style="height: 1em; max-height: 1em;" />even former developers<img src=—" class="wp-smiley" style="height: 1em; max-height: 1em;" />he hates context switching, and that's what he found himself doing as he moved from application to application as he went about his day to day duties as a CEO. Each application had it's own UI and its own set of shortcut keys. Recently, he decided to revisit Emacs and to try to do as many of his daily tasks as possible from within Emacs.

Stella describes his new set up and writes about why other CEOs might want to try Emacs too. It's not for everyone, he admits, but if you're the right sort of person, Emacs can revolutionize your work flow and make you more efficient and happier. None of that will come as a surprise to us Emacsers, of course, but I wonder how many CEOs without a technical background will be willing to climb up the learning curve to get those benefits.

To make that climb a bit easier, Stella spends some time describing how to install Emacs and goes over some of the basic navigation. If you're looking for reasons why a non-technical person might want to try Emacs, Stella's post is an excellent place to start.

-1:-- Emacs for the CEO (Post jcs)--L0--C0--November 25, 2015 05:11 PM

emacspeak: Listening To Multiple Media Streams On The Emacspeak Audio Desktop

Listening To Multiple Media Streams On The Emacspeak Audio Desktop

1 Executive Summary

The GitHub version of Emacspeak now supports launching and controlling
multiple media streams. This enables one to listen to the news while
playing a music stream, or relaxing nature sounds.

2 Sample Usage

Here are some examples of using this feature:

  1. Launch your favorite news station — BBC World Service in my
    case — C-e ; RET.
  2. Place the News on the left channel — C-e ; (.
  3. Persist the currently playing News stream by invoking command
    emacspeak-m-player-persist-stream bound to C-e ; \. This lets
    you launch a second stream via Emacspeak media key C-e ; rather
    than controlling the currently playing stream.
  4. Launch a classical music media-stream — C-e ; lu RET for a lullaby
    media stream.
  5. Now Emacspeak M-Player commands will control the most recently
    launched stream; you can once again invoke command
    emacspeak-m-player-persist-stream if you wish.
  6. The previously launched (and still playing) News stream is now in a
    buffer named *Persistent-...*. Command
    emacspeak-wizards-view-buffers-filtered-by-m-player-mode can be
    used to list buffers that hold a live m-player instance. It is
    bound to b in emacspeak-m-player-mode. I also bind this command
    to C-; ; in my global keymap.
  7. You can make an M-Player instance current by switching to its
    buffer and invoking command emacspeak-m-player-restore-process
    bound to / in emacspeak-m-player-mode.

Share And Enjoy–

-1:-- Listening To Multiple Media Streams On The Emacspeak Audio Desktop (Post T. V. Raman ( 23, 2015 11:25 PM

Matt Price: PDF Tools Post amended

I’ve updated my older post on PDF Tools with slightly cleaner code, for anyone who’s using this method of extracting annotations!

-1:-- PDF Tools Post amended (Post matt)--L0--C0--November 23, 2015 04:43 PM

sachachua: 2015-11-23 Emacs News

Links from, Hacker News,, Youtube, the Emacs commit log, the changes to the Emacs NEWS file, and emacs-devel

Past Emacs News round-ups

The post 2015-11-23 Emacs News appeared first on sacha chua :: living an awesome life.

-1:-- 2015-11-23 Emacs News (Post Sacha Chua)--L0--C0--November 23, 2015 04:21 PM

Ryan Rix: Body Computing System continuations

Over the last few weeks, I have integrated the Body Computing System more and more deeply in to my workflow and moving more and more systems out of my smartphone and shitty decentralized systems.
-1:-- Body Computing System continuations (Post)--L0--C0--November 23, 2015 12:00 AM

Endless Parentheses: Update on tdd-mode with CIDER

I can’t write a whole new post this week due to being buried under some once-in-a-lifetime stuff. Still, the Monday post is one I refuse to miss, so I leave you today with an update on last weeks post.

Running tests on file save really didn’t work out for me, because I’ve configured CIDER to save files before loading them when I use cider-load-file. So the tests were always being run against outdated definitions.

The fix is simple, though. Just run the tests after load-file. CIDER even had a hook for that already. I also made the minor-mode global (even though it only applies to clojure buffers) so that it’s easier to disable everywhere with a single command.

The new code is in the original post.

Comment on this.

-1:-- Update on tdd-mode with CIDER (Post)--L0--C0--November 23, 2015 12:00 AM

Matt Price: Note Taking with PDF Tools

NOTE: This post has been modified as of 2015-11-22 Sun – the new code is a little cleaner, and I think the discussion a little fuller.

Almost all of my job-related reading is now done on a screen. There are still disadvantages – I find it much harder to concentrate when reading online – but in other ways it is markedly more convenient.

In particular, it is now much easier to assemble quotations from sources; and now that I’ve found PDF Tools, it has become even easier. I’ve just started to use it to extract annotations from my PDF’s, and it works much better than the lousy command-line hack I was using previously.

As we’re mid-semester, most of my reading is for classes I teach. My current workflow is as follows:

  • Assemble the relevant readings in a Dropbox-synced directory (ClassName/Readings)
  • Using Repligo Reader (apparently no longer available in the app store?), highlight the passages I’m interested in.
  • execute code block (see below) to insert org headings with all highlights from one or more pdfs
  • Assemble reveal.js lecture presentation around those highlights, using org-reveal or Pandoc.

Activating PDF Tools

Begin by installing pdf-tools and org-pdfview from ELPA with package-list~packages or package-install.

Then make sure they are activated by adding these lines in your init file:

(eval-after-load 'org '(require 'org-pdfview))
(add-to-list 'org-file-apps '("\\.pdf\\'" . org-pdfview-open))
(add-to-list 'org-file-apps '("\\.pdf::\\([[:digit:]]+\\)\\'" . org-pdfview-open))

Switching to PDF Tools for annotating and extracting PDF’s

Last month Penguim proposed some changes in a pull request, that export annotations as a set of org headlines. It’s potentially very interesting but not quite what I want to do, so I modified this code. pdf-annot-markups-as-org-text extracts the text of an annotation (stored as the subject attribute in an alist), and also generates a link back to the page in the pdf. mwp/pdf-multi-extract is just a helper function that makes it easier to construct elisp source blocks the way I’m used to doing:

;; modified from 

(defun mwp/pdf-multi-extract (sources)
  "Helper function to print highlighted text from a list of pdf's, with one org header per pdf, 
and links back to page of highlight."
  (let (
        (output ""))
    (dolist (thispdf sources)
      (setq output (concat output (pdf-annot-markups-as-org-text thispdf nil level ))))
    (princ output))

;; this is stolen from
(defun pdf-annot-edges-to-region (edges)
  "Attempt to get 4-entry region \(LEFT TOP RIGHT BOTTOM\) from several edges.
We need this to import annotations and to get marked-up text, because annotations
are referenced by its edges, but functions for these tasks need region."

  (let ((left0 (nth 0 (car edges)))
        (top0 (nth 1 (car edges)))
        (bottom0 (nth 3 (car edges)))
        (top1 (nth 1 (car (last edges))))
        (right1 (nth 2 (car (last edges))))
        (bottom1 (nth 3 (car (last edges))))
        (n (safe-length edges)))
    ;; we try to guess the line height to move
    ;; the region away from the boundary and
    ;; avoid double lines
    (list left0
          (+ top0 (/ (- bottom0 top0) 2))
          (- bottom1 (/ (- bottom1 top1) 2 )))))

(defun pdf-annot-markups-as-org-text (pdfpath &optional title level)
  "Acquire highligh annotations as text, and return as org-heading"

  (interactive "fPath to PDF: ")  
  (let* ((outputstring "") ;; the text to be returned
         (title (or title (replace-regexp-in-string "-" " " (file-name-base pdfpath ))))
         (level (or level (1+ (org-current-level)))) ;; I guess if we're not in an org-buffer this will fail
         (levelstring (make-string level ?*)) ;; set headline to proper level
         (annots (sort (pdf-info-getannots nil pdfpath)  ;; get and sort all annots
    ;; create the header
    (setq outputstring (concat levelstring " Quotes From " title "\n\n")) ;; create heading

    ;; extract text
     (lambda (annot) ;; traverse all annotations
       (if (eq 'highlight (assoc-default 'type annot))
           (let* ((page (assoc-default 'page annot))
                  ;; use pdf-annot-edges-to-region to get correct boundaries of highlight
                  (real-edges (pdf-annot-edges-to-region
                               (pdf-annot-get annot 'markup-edges)))
                  (text (or (assoc-default 'subject annot) (assoc-default 'content annot)
                            (replace-regexp-in-string "\n" " " (pdf-info-gettext page real-edges nil pdfpath)
                                                      ) ))

                  (height (nth 1 real-edges)) ;; distance down the page
                  ;; use pdfview link directly to page number
                  (linktext (concat "[[pdfview:" pdfpath "::" (number-to-string page) 
                                    "++" (number-to-string height) "][" title  "]]" ))
             (setq outputstring (concat outputstring text " ("
                                        linktext ", " (number-to-string page) ")\n\n"))
    outputstring ;; return the header

Using in Org with a Source Block

Now it’s more or less trivial to quickly generate the org headers using a source block:

#+BEGIN_SRC elisp :results output raw :var level=(1+ (org-current-level))
(mwp/pdf-multi-extract '(
                   "/home/matt/HackingHistory/readings/Troper-becoming-immigrant-city.pdf"  "/home/matt/HackingHistory/readings/historical-authority-hampton.pdf"))


And the output gives something like


Quotes From Troper becoming immigrant city

Included in the Greater Toronto Area multiethnic mix are an estimated 450,000 Chinese, 400,000 Italians, and 250,000 African Canadians, the largest component of which are ofCar- ibbean background, although a separate and distinct infusion of Soma- lis, Ethiopians, and other Africans is currently taking place. (Troper becoming immigrant city, 3)

Although Toronto is Canada’s leading immigrant-receiving centre, city officials have neither a hands-on role in immigrant selection nor an official voice in deciding immigration policy. In Canada, immigration policy and administration is a constitutional responsibility of the fed- eral government, worked out in consultation with the provinces. (Troper becoming immigrant city, 4)


Alternative: Temporary buffer with custom link type

An alternative workflow would be to pop to a second, temporary buffer and insert the annotations there; one could do this with a custom link type. PDF-Tools already has a mechanism for listing annotations in a separate buffer, but it’s not designed for quick access to all annotations at once. Anyway, here’s one way to do this; I’m not really using it at the moment.

(org-add-link-type "pdfquote" 'org-pdfquote-open 'org-pdfquote-export)

(defun org-pdfquote-open (link)
  "Open a new buffer with all markup annotations in an org headline."
   (format "*Quotes from %s*"
           (file-name-base link)))
  (insert (pdf-annot-markups-as-org-text link nil 1))
  (goto-char 0)

(defun org-pdfquote-export (link description format)
  "Export the pdfview LINK with DESCRIPTION for FORMAT from Org files."
  (let* ((path (when (string-match "\\(.+\\)::.+" link)
                 (match-string 1 link)))
         (desc (or description link)))
    (when (stringp path)
      (setq path (org-link-escape (expand-file-name path)))
       ((eq format 'html) (format "<a href=\"%s\">%s</a>" path desc))
       ((eq format 'latex) (format "\href{%s}{%s}" path desc))
       ((eq format 'ascii) (format "%s (%s)" desc path))
       (t path)))))

(defun org-pdfquote-complete-link ()
  "Use the existing file name completion for file.
Links to get the file name, then ask the user for the page number
and append it."                                  

  (replace-regexp-in-string "^file:" "pdfquote:" (org-file-complete-link)))

I’ve also added two bindings to make highlighting easier from the PDF buffer:

(eval-after-load 'pdf-view 
                    '(define-key pdf-view-mode-map (kbd "M-h") 'pdf-annot-add-highlight-markup-annotation))
(eval-after-load 'pdf-view 
                    '(define-key pdf-view-mode-map (kbd "<tab>") 'pdf-annot-add-highlight-markup-annotation))

All of this is getting me very close to using Emacs for all my PDF work. I am doing maybe 50% of my PDF work in Emacs instead of on my tablet. It’s incredibly convenient, although I still find it a little harder to concentrate on my laptop than on the tablet (for reasons ergonomic, optical, and psychological). Here are the remaining papercuts from my perspective:

  • Highlighting text with the mouse is more awkward and less intimate than using my fingertip on a laptop. I often find mouse movement a little awkward in Emacs, but pdf-view purposely relies on the mouse for movement (for good reasons).
  • Scrolling in pdf-view is also a bit awkward, and there’s no “continuous” mode as one might find in Evince or acroread. Again, I often find scrolling an issue in Emacs, so this might not be so easy to fix.
  • Finally, the laptop screen is just harder on my eyes than my high-res tablet. pdf-view hasa “midnight mode” which makes it a little easier to read, but it’s not quite enough.

So, for the time being I will probably do much of my reading on the tablet. But for short pieces and for review (e.g., papers that I’m reading for the third year in a row in a graduate seminar…) PDF Tools is now my main interface. Which is pretty sweet.


UPDATE: I would like to extend the pdfview link type (in org-pdfview) to permit me to specify the precise location of an annotation, so I can jump precisely to that part of the page. This has now been done and the code above has been updated to reflect the new syntax.

UPDATE: Also, now that I think about it, it might be interesting to just have a link type that pops up a temporary buffer with all of the annotations; I could then cut and paste the annotations into the master document. This might be even more convenient. OK, I’ve implemented this, see above!

Update 2015-11-22 Sun: I’ve cleaned up some of the code, and added a bit more commentary at the end.

-1:-- Note Taking with PDF Tools (Post matt)--L0--C0--November 22, 2015 09:25 PM

Marcin Borkowski: The Emacs widget library and automatic modification of editing fields

A long time ago, I read about the Emacs Widget library. My first thought was “Wow, that is so cool!”. I didn’t have any use for that, however. Until recently.
-1:-- The Emacs widget library and automatic modification of editing fields (Post)--L0--C0--November 22, 2015 12:47 AM

Irreal: Gnus on Windows

As most Emacers know, two of the most difficult things involving Emacs are

  1. Getting it to work well on Windows, and
  2. Configuring and using Gnus.

So, of course, Sacha Chua decided to do both at the same time.

In the above post, Chua goes through the configuration she used to get it all working. Since she keeps her Emacs configuration as an Org file, she just cut the Gnus part right out of the configuration file and pasted it into her post. Yet another benefit of keeping your configuration as an Org file.

If you're working on Windows and want to try out Gnus, take a look at Chua's post. She uses Gmail so you can also see how to get that working with Gnus. Like all of her posts, there's a lot of meat in it so it's worth taking a look even if you aren't interested in Gnus at the moment.


-1:-- Gnus on Windows (Post jcs)--L0--C0--November 21, 2015 06:10 PM

Raimon Grau: Mu4e and gmail (multiple accounts)

Using Emacs for reading mails is super neat and all, but honestly, gmail does an amazing job managing mails.  To make things even 'worse', I wrote some small js to ease the process of managing multiple gmail accounts in conkeror.

I tried to replace gmail with gnus multiple times, but it never worked out very well. Mu4e seems the most sensible choice, but I always had issues configuring it for my gmail accounts ("Your IMAP server wants to alert you to the following: Please log in via your web browser: (Failure)" being the last one), but I finally got to solve them.

If you get the "Please log in via your web browser" there are 2 ways of solving it.
  1. As explained here, allow less secure apps, by going to your accouts settings. This may not work in your company account (if they disabled this option)
  2. Add an application password token. By going to your security settings. and adding a new 'application password'. You can use that token in your .offlineimaprc  file.

Once the basic configuration was working, and read pretty much all the docs and blogs on the field, I quickly got annoyed by the 'dashboard' view mu4e offers.  It provides exactly 0 useful info.  So I got rolled up my sleeves and wrote this tiny little function to do my usual actions in mu4e:

I want to know about new mails in both my personal and corporate accounts.

(defun mnew ()
(mu4e-headers-search-bookmark "flag:unread AND NOT flag:trashed AND maildir:/ OR
flag:unread AND NOT flag:trashed AND maildir:/"))

Not very nice, I know, but for now, it already provides me something I wanted and wasn't quite possible in gmail. also, I don't have to visit the mu4e 'dashboard'.

Ah!, to periodically fetch new mails, for now I'm going with elisp's timer `run-at-time'.

(run-at-time nil (* 60 5) 'mu4e-update-mail-and-index t)

-1:-- Mu4e and gmail (multiple accounts) (Post Raimon Grau ( 21, 2015 11:59 AM

Chris Wellons: Quickly Access x86 Documentation in Emacs

I recently released an Emacs package called x86-lookup. Given a mnemonic, Emacs will open up a local copy of an Intel’s software developer manual PDF at the page documenting the instruction. It complements nasm-mode, released earlier this year.

x86-lookup is also available from MELPA.

To use it, you’ll need Poppler’s pdftotext command line program — used to build an index of the PDF — and a copy of the complete Volume 2 of Intel’s instruction set manual. There’s only one command to worry about: M-x x86-lookup.

Minimize documentation friction

This package should be familiar to anyone who’s used javadoc-lookup, one of my older packages. It has a common underlying itch: the context switch to read API documentation while coding should have as little friction as possible, otherwise I’m discouraged from doing it. In an ideal world I wouldn’t ever need to check documentation because it’s already in my head. By visiting documentation frequently with ease, it’s going to become familiar that much faster and I’ll be reaching for it less and less, approaching the ideal.

I picked up x86 assembly [about a year ago][x86] and for the first few months I struggled to find a good online reference for the instruction set. There are little scraps here and there, but not much of substance. The big exception is Félix Cloutier’s reference, which is an amazingly well-done HTML conversion of Intel’s PDF manuals. Unfortunately I could never get it working locally to generate my own. There’s also the X86 Opcode and Instruction Reference, but it’s more for machines than humans.

Besides, I often work without an Internet connection, so offline documentation is absolutely essential. (You hear that Microsoft? Not only do I avoid coding against Win32 because it’s badly designed, but even more so because you don’t offer offline documentation anymore! The friction to API reference your documentation is enormous.)

I avoided the official x86 documentation for awhile, thinking it would be too opaque, at least until I became more accustomed to the instruction set. But really, it’s not bad! With a handle on the basics, I would encourage anyone to dive into either Intel’s or AMD’s manuals. The reason there’s not much online in HTML form is because these manuals are nearly everything you need.

I chose Intel’s manuals for x86-lookup because I’m more familiar with it, it’s more popular, it’s (slightly) easier to parse, it’s offered as a single PDF, and it’s more complete. The regular expression for finding instructions is tuned for Intel’s manual and it won’t work with AMD’s manuals.

For a couple months prior to writing x86-lookup, I had a couple of scratch functions to very roughly accomplish the same thing. The tipping point for formalizing it was that last month I wrote my own x86 assembler. A single mnemonic often has a dozen or more different opcodes depending on the instruction’s operands, and there are often several ways to encode the same operation. I was frequently looking up opcodes, and navigating the PDF quickly became a real chore. I only needed about 80 different opcodes, so I was just adding them to the assembler’s internal table manually as needed.

How does it work?

Say you want to look up the instruction RDRAND.

Initially Emacs has no idea what page this is on, so the first step is to build an index mapping mnemonics to pages. x86-lookup runs the pdftotext command line program on the PDF and loads the result into a temporary buffer.

The killer feature of pdftotext is that it emits FORM FEED (U+0012) characters between pages. Think of these as page breaks. By counting form feed characters, x86-lookup can track the page for any part of the document. In fact, Emacs is already set up to do this with its forward-page and backward-page commands. So to build the index, x86-lookup steps forward page-by-page looking for mnemonics, keeping note of the page. Since this process typically takes about 10 seconds, the index is cached in a file (see x86-lookup-cache-directory) for future use. It only needs to happen once for a particular manual on a particular computer.

The mnemonic listing is slightly incomplete, so x86-lookup expands certain mnemonics into the familiar set. For example, all the conditional jumps are listed under “Jcc,” but this is probably not what you’d expect to look up. I compared x86-lookup’s mnemonic listing against NASM/nasm-mode’s mnemonics to ensure everything was accounted for. Both packages benefited from this process.

Once the index is built, pdftotext is no longer needed. If you’re desperate and don’t have this program available, you can borrow the index file from another computer. But you’re on your own for figuring that out!

So to look up RDRAND, x86-lookup checks the index for the page number and invokes a PDF reader on that page. This is where not all PDF readers are created equal. There’s no convention for opening a PDF to a particular page and each PDF reader differs. Some don’t even support it. To deal with this, x86-lookup has a function specialized for different PDF readers. Similar to browse-url-browser-function, x86-lookup has x86-lookup-browse-pdf-function.

By default it tries to open the PDF for viewing within Emacs (did you know Emacs is a PDF viewer?), falling back to on options if the feature is unavailable. I welcome pull requests for any PDF readers not yet supported by x86-lookup. Perhaps this functionality deserves its own package.

That’s it! It’s a simple feature that has already saved me a lot of time. If you’re ever programming in x86 assembly, give x86-lookup a spin.

-1:-- Quickly Access x86 Documentation in Emacs (Post)--L0--C0--November 21, 2015 05:42 AM

emacspeak: Emacspeak 43.0 (SoundDog) Unleashed!

Emacspeak 43.0—SoundDog—Unleashed!

For Immediate Release:

San Jose, Calif., (Nov 20, 2015)
Emacspeak: Redefining Accessibility In The Era Of Internet Computing
–Zero cost of Ownership makes priceless software Universally affordable!

Emacspeak Inc (NASDOG: ESPK) --– announces the
immediate world-wide availability of Emacspeak 43.0 (SoundDog) –a
powerful audio desktop for leveraging today's evolving data, social
and service-oriented Internet cloud.

1 Investors Note:

With several prominent tweeters expanding coverage of
#emacspeak, NASDOG: ESPK has now been consistently trading over
the social net at levels close to that once attained by DogCom
high-fliers—and as of Nov 2015 is trading at levels close to
that achieved by once better known stocks in the tech sector.

2 What Is It?

Emacspeak is a fully functional audio desktop that provides
complete eyes-free access to all major 32 and 64 bit operating
environments. By seamlessly blending live access to all aspects
of the Internet such as Web-surfing, blogging, social computing
and electronic messaging into the audio desktop, Emacspeak
enables speech access to local and remote information with a
consistent and well-integrated user interface. A rich suite of
task-oriented tools provides efficient speech-enabled access to
the evolving service-oriented social Internet cloud.

3 Major Enhancements:

  • Multiple spatially located TTS streams for efficient communication. 🕪
  • Refactored, improved sound themes.℗
  • Updated org-mode support. 🎶
  • Speech-enables package helm.𝍎
  • Speech-enable package yasnippet for smart templates. ䷾
  • Context-sensitive keyboard shortcuts via package emacspeak-muggles. ⌨
  • Speech-enables lua-mode for LUA programming. Ÿ
  • Speech-enable package projectile for software project

management. 📽

  • Speech-enable package slime for Lisp programming. Λ
  • Updated Librivox support to the new API. 📖
  • Package XBacklight for controlling LCD brightness. ⎚
  • Updated NPR client for one-click access to present and past NPR content.📻
  • BBC radio programs via IPlayer and friends. 📢
  • Emacs EWW: Consume Web content efficiently. 🕷
  • Updated Info manual 🕮
  • Speech-enabled Elfeed, an Emacs Feed Reader 🗞
  • emacspeak-url-templates: Smart Web access. ♅
  • emacspeak-websearch.el Find things fast. ♁
  • And a lot more than wil fit this margin. …

4 Establishing Liberty, Equality And Freedom:

Never a toy system, Emacspeak is voluntarily bundled with all
major Linux distributions. Though designed to be modular,
distributors have freely chosen to bundle the fully integrated
system without any undue pressure—a documented success for
the integrated innovation embodied by Emacspeak. As the system
evolves, both upgrades and downgrades continue to be available at
the same zero-cost to all users. The integrity of the Emacspeak
codebase is ensured by the reliable and secure Linux platform
used to develop and distribute the software.

Extensive studies have shown that thanks to these features, users
consider Emacspeak to be absolutely priceless. Thanks to this
wide-spread user demand, the present version remains priceless
as ever—it is being made available at the same zero-cost as
previous releases.

At the same time, Emacspeak continues to innovate in the area of
eyes-free social interaction and carries forward the
well-established Open Source tradition of introducing user
interface features that eventually show up in luser environments.

On this theme, when once challenged by a proponent of a crash-prone
but well-marketed mousetrap with the assertion "Emacs is a system from
the 70's", the creator of Emacspeak evinced surprise at the unusual
candor manifest in the assertion that it would take popular
idiot-proven interfaces until the year 2070 to catch up to where the
Emacspeak audio desktop is today. Industry experts welcomed this
refreshing breath of Courage Certainty and Clarity (CCC) at a time
when users are reeling from the Fear Uncertainty and Doubt (FUD)
unleashed by complex software systems backed by even more convoluted
press releases.

5 Independent Test Results:

Independent test results have proven that unlike some modern (and
not so modern) software, Emacspeak can be safely uninstalled without
adversely affecting the continued performance of the computer. These
same tests also revealed that once uninstalled, the user stopped
functioning altogether. Speaking with Aster Labrador, the creator of
Emacspeak once pointed out that these results re-emphasize the
user-centric design of Emacspeak; "It is the user –and not the
computer– that stops functioning when Emacspeak is uninstalled!".

5.1 Note from Aster,Bubbles and Tilden:

UnDoctored Videos Inc. is looking for volunteers to star in a
video demonstrating such complete user failure.

6 Obtaining Emacspeak:

Emacspeak can be downloaded from GitHub –see you can visit
Emacspeak on the WWW at You can subscribe
to the emacspeak mailing list by sending
mail to the list request address The
Emacspeak Blog is a good source
for news about recent enhancements and how to use them.

The latest development snapshot of Emacspeak is always available via
Git from GitHub at
Emacspeak GitHub .

7 History:

  • Emacspeak 43.0 brings even more end-user efficiency by leveraging the
    ability to spatially place multiple audio streams to provide timely
    auditory feedback.
  • Emacspeak 42.0 while moving to GitHub from Google Code continues to
    innovate in the areas of auditory user interfaces and efficient,
    light-weight Internet access.
  • Emacspeak 41.0 continues to improve
    on the desire to provide not just equal, but superior access —
    technology when correctly implemented can significantly enhance the
    human ability.
  • Emacspeak 40.0 goes back to Web basics by enabling
    efficient access to large amounts of readable Web content.
  • Emacspeak 39.0 continues the Emacspeak tradition of increasing the breadth of
    user tasks that are covered without introducing unnecessary
  • Emacspeak 38.0 is the latest in a series of award-winning
    releases from Emacspeak Inc.
  • Emacspeak 37.0 continues the tradition of
    delivering robust software as reflected by its code-name.
  • Emacspeak 36.0 enhances the audio desktop with many new tools including full
    EPub support — hence the name EPubDog.
  • Emacspeak 35.0 is all about
    teaching a new dog old tricks — and is aptly code-named HeadDog in
    on of our new Press/Analyst contact. emacspeak-34.0 (AKA Bubbles)
    established a new beach-head with respect to rapid task completion in
    an eyes-free environment.
  • Emacspeak-33.0 AKA StarDog brings
    unparalleled cloud access to the audio desktop.
  • Emacspeak 32.0 AKA
    LuckyDog continues to innovate via open technologies for better
  • Emacspeak 31.0 AKA TweetDog — adds tweeting to the Emacspeak
  • Emacspeak 30.0 AKA SocialDog brings the Social Web to the
    audio desktop—you cant but be social if you speak!
  • Emacspeak 29.0—AKAAbleDog—is a testament to the resilliance and innovation
    embodied by Open Source software—it would not exist without the
    thriving Emacs community that continues to ensure that Emacs remains
    one of the premier user environments despite perhaps also being one of
    the oldest.
  • Emacspeak 28.0—AKA PuppyDog—exemplifies the rapid pace of
    development evinced by Open Source software.
  • Emacspeak 27.0—AKA
    FastDog—is the latest in a sequence of upgrades that make previous
    releases obsolete and downgrades unnecessary.
  • Emacspeak 26—AKA
    LeadDog—continues the tradition of introducing innovative access
    solutions that are unfettered by the constraints inherent in
    traditional adaptive technologies.
  • Emacspeak 25 —AKA ActiveDog
    —re-activates open, unfettered access to online
  • Emacspeak-Alive —AKA LiveDog —enlivens open, unfettered
    information access with a series of live updates that once again
    demonstrate the power and agility of open source software
  • Emacspeak 23.0 – AKA Retriever—went the extra mile in
    fetching full access.
  • Emacspeak 22.0 —AKA GuideDog —helps users
    navigate the Web more effectively than ever before.
  • Emacspeak 21.0
    —AKA PlayDog —continued the
    Emacspeak tradition of relying on enhanced
    productivity to liberate users.
  • Emacspeak-20.0 —AKA LeapDog —continues
    the long established GNU/Emacs tradition of integrated innovation to
    create a pleasurable computing environment for eyes-free
  • emacspeak-19.0 –AKA WorkDog– is designed to enhance
    user productivity at work and leisure.
  • Emacspeak-18.0 –code named
    GoodDog– continued the Emacspeak tradition of enhancing user
    productivity and thereby reducing total cost of
  • Emacspeak-17.0 –code named HappyDog– enhances user
    productivity by exploiting today's evolving WWW
  • Emacspeak-16.0 –code named CleverDog– the follow-up to
    SmartDog– continued the tradition of working better, faster,
  • Emacspeak-15.0 –code named SmartDog–followed up on TopDog
    as the next in a continuing series of award-winning audio desktop
    releases from Emacspeak Inc.
  • Emacspeak-14.0 –code named TopDog–was

the first release of this millennium.

  • Emacspeak-13.0 –codenamed
    YellowLab– was the closing release of the
    20th. century.
  • Emacspeak-12.0 –code named GoldenDog– began
    leveraging the evolving semantic WWW to provide task-oriented speech
    access to Webformation.
  • Emacspeak-11.0 –code named Aster– went the
    final step in making Linux a zero-cost Internet access solution for
    blind and visually impaired users.
  • Emacspeak-10.0 –(AKA
    Emacspeak-2000) code named WonderDog– continued the tradition of
    award-winning software releases designed to make eyes-free computing a
    productive and pleasurable experience.
  • Emacspeak-9.0 –(AKA
    Emacspeak 99) code named BlackLab– continued to innovate in the areas
    of speech interaction and interactive accessibility.
  • Emacspeak-8.0 –(AKA Emacspeak-98++) code named BlackDog– was a major upgrade to
    the speech output extension to Emacs.
  • Emacspeak-95 (code named Illinois) was released as OpenSource on
    the Internet in May 1995 as the first complete speech interface
    to UNIX workstations. The subsequent release, Emacspeak-96 (code
    named Egypt) made available in May 1996 provided significant
    enhancements to the interface. Emacspeak-97 (Tennessee) went
    further in providing a true audio desktop. Emacspeak-98
    integrated Internetworking into all aspects of the audio desktop
    to provide the first fully interactive speech-enabled WebTop.

8 About Emacspeak:

Originally based at Cornell (NY) –home to Auditory User
Interfaces (AUI) on the WWW– Emacspeak is now maintained on
GitHub -- —and
Sourceforge — The system is mirrored
world-wide by an international network of software archives and
bundled voluntarily with all major Linux distributions. On
Monday, April 12, 1999, Emacspeak became part of the
Smithsonian's Permanent Research Collection on Information
Technology at the Smithsonian's National Museum of American

The Emacspeak mailing list is archived at Vassar –the home of the
Emacspeak mailing list– thanks to Greg Priest-Dorman, and provides a
valuable knowledge base for new users.

9 Press/Analyst Contact: Tilden Labrador

Going forward, Tilden acknowledges his exclusive monopoly on
setting the direction of the Emacspeak Audio Desktop, and
promises to exercise this freedom to innovate and her resulting
power responsibly (as before) in the interest of all dogs.

*About This Release:

Windows-Free (WF) is a favorite battle-cry of The League Against
Forced Fenestration (LAFF). –see for details on
the ill-effects of Forced Fenestration.

CopyWrite )C( Aster and Hubbell Labrador. All Writes Reserved.
HeadDog (DM), LiveDog (DM), GoldenDog (DM), BlackDog (DM) etc., are Registered
Dogmarks of Aster, Hubbell and Tilden Labrador. All other dogs belong to
their respective owners.

-1:-- Emacspeak 43.0 (SoundDog) Unleashed! (Post T. V. Raman ( 20, 2015 06:07 PM

Julien Danjou: Profiling Python using cProfile: a concrete case

Writing programs is fun, but making them fast can be a pain. Python programs are no exception to that, but the basic profiling toolchain is actually not that complicated to use. Here, I would like to show you how you can quickly profile and analyze your Python code to find what part of the code you should optimize.

What's profiling?

Profiling a Python program is doing a dynamic analysis that measures the execution time of the program and everything that compose it. That means measuring the time spent in each of its functions. This will give you data about where your program is spending time, and what area might be worth optimizing.

It's a very interesting exercise. Many people focus on local optimizations, such as determining e.g. which of the Python functions range or xrange is going to be faster. It turns out that knowing which one is faster may never be an issue in your program, and that the time gained by one of the functions above might not be worth the time you spend researching that, or arguing about it with your colleague.

Trying to blindly optimize a program without measuring where it is actually spending its time is a useless exercise. Following your guts alone is not always sufficient.

There are many types of profiling, as there are many things you can measure. In this exercise, we'll focus on CPU utilization profiling, meaning the time spent by each function executing instructions. Obviously, we could do many more kind of profiling and optimizations, such as memory profiling which would measure the memory used by each piece of code – something I talk about in The Hacker's Guide to Python.


Since Python 2.5, Python provides a C module called cProfile which has a reasonable overhead and offers a good enough feature set. The basic usage goes down to:

>>> import cProfile
>>>'2 + 2')
2 function calls in 0.000 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}

Though you can also run a script with it, which turns out to be handy:

$ python -m cProfile -s cumtime
72270 function calls (70640 primitive calls) in 4.481 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.004 0.004 4.481 4.481<module>)
1 0.001 0.001 4.296 4.296
3 0.000 0.000 4.286 1.429
3 0.000 0.000 4.268 1.423
4/3 0.000 0.000 3.816 1.272
4 0.000 0.000 2.965 0.741
4 0.000 0.000 2.962 0.740
4 0.000 0.000 2.961 0.740
2 0.000 0.000 2.675 1.338
30 0.000 0.000 1.621 0.054
30 0.000 0.000 1.621 0.054
30 1.621 0.054 1.621 0.054 {method 'read' of '_ssl._SSLSocket' objects}
1 0.000 0.000 1.611 1.611
4 0.000 0.000 1.572 0.393
4 0.000 0.000 1.572 0.393
60 0.000 0.000 1.571 0.026
4 0.000 0.000 1.571 0.393
1 0.000 0.000 1.462 1.462
1 0.000 0.000 1.462 1.462
1 0.000 0.000 1.462 1.462
1 0.000 0.000 1.459 1.459

This prints out all the function called, with the time spend in each and the number of times they have been called.

Advanced visualization with KCacheGrind

While being useful, the output format is very basic and does not make easy to grab knowledge for complete programs. For more advanced visualization, I leverage KCacheGrind. If you did any C programming and profiling these last years, you may have used it as it is primarily designed as front-end for Valgrind generated call-graphs.

In order to use, you need to generate a cProfile result file, then convert it to KCacheGrind format. To do that, I use pyprof2calltree.

$ python -m cProfile -o myscript.cprof
$ pyprof2calltree -k -i myscript.cprof

And the KCacheGrind window magically appears!

Concrete case: Carbonara optimization

I was curious about the performances of Carbonara, the small timeserie library I wrote for Gnocchi. I decided to do some basic profiling to see if there was any obvious optimization to do.

In order to profile a program, you need to run it. But running the whole program in profiling mode can generate a lot of data that you don't care about, and adds noise to what you're trying to understand. Since Gnocchi has thousands of unit tests and a few for Carbonara itself, I decided to profile the code used by these unit tests, as it's a good reflection of basic features of the library.

Note that this is a good strategy for a curious and naive first-pass profiling. There's no way that you can make sure that the hotspots you will see in the unit tests are the actual hotspots you will encounter in production. Therefore, a profiling in conditions and with a scenario that mimics what's seen in production is often a necessity if you need to push your program optimization further and want to achieve perceivable and valuable gain.

I activated cProfile using the method described above, creating a cProfile.Profile object around my tests (I actually started to implement that in testtools). I then run KCacheGrind as described above. Using KCacheGrind, I generated the following figures.

The test I profiled here is called test_fetch and is pretty easy to understand: it puts data in a timeserie object, and then fetch the aggregated result. The above list shows that 88 % of the ticks are spent in set_values (44 ticks over 50). This function is used to insert values into the timeserie, not to fetch the values. That means that it's really slow to insert data, and pretty fast to actually retrieve them.

Reading the rest of the list indicates that several functions share the rest of the ticks, update, _first_block_timestamp, _truncate, _resample, etc. Some of the functions in the list are not part of Carbonara, so there's no point in looking to optimize them. The only thing that can be optimized is, sometimes, the number of times they're called.

The call graph gives me a bit more insight about what's going on here. Using my knowledge about how Carbonara works, I don't think that the whole stack on the left for _first_block_timestamp makes much sense. This function is supposed to find the first timestamp for an aggregate, e.g. with a timestamp of 13:34:45 and a period of 5 minutes, the function should return 13:30:00. The way it works currently is by calling the resample function from Pandas on a timeserie with only one element, but that seems to be very slow. Indeed, currently this function represents 25 % of the time spent by set_values (11 ticks on 44).

Fortunately, I recently added a small function called _round_timestamp that does exactly what _first_block_timestamp needs that without calling any Pandas function, so no resample. So I ended up rewriting that function this way:

def _first_block_timestamp(self):
- ts = self.ts[-1:].resample(self.block_size)
- return (ts.index[-1] - (self.block_size * self.back_window))
+ rounded = self._round_timestamp(self.ts.index[-1], self.block_size)
+ return rounded - (self.block_size * self.back_window)

And then I re-run the exact same test to compare the output of cProfile.

The list of function seems quite different this time. The number of time spend used by set_values dropped from 88 % to 71 %.

The call stack for set_values shows that pretty well: we can't even see the _first_block_timestamp function as it is so fast that it totally disappeared from the display. It's now being considered insignificant by the profiler.

So we just speed up the whole insertion process of values into Carbonara by a nice 25 % in a few minutes. Not that bad for a first naive pass, right?

-1:-- Profiling Python using cProfile: a concrete case (Post Julien Danjou)--L0--C0--November 16, 2015 03:55 PM

Endless Parentheses: Test-Driven-Development in CIDER and Emacs

As I was catching up on a few Parens of the Dead episodes this weekend, I was amused at how Magnar set up his Emacs to run tests whenever the file is saved. At first I thought it wasn’t for me (I’m one of those who obsessively saves every few seconds), but I’ve been trying it out lately and it’s starting to grow on me.

At its core, all you really need is an hook. But I walked the extra yard and wrote a minor mode for it, so I can easily call M-x tdd-mode to disable it if it ever gets on my nerves.

(defun tdd-test ()
  "Thin wrapper around `cider-test-run-tests'."
  (when (cider-connected-p)
    (let ((cider-auto-select-test-report-buffer nil)
          (cider-test-show-report-on-success nil))
      (cider-test-run-tests nil))))

(define-minor-mode tdd-mode
  "Run all tests whenever a file is saved."
  t nil nil
  :global t
  (if tdd-mode
      (add-hook 'cider-file-loaded-hook #'tdd-test)
    (remove-hook 'cider-file-loaded-hook #'tdd-test)))

I also had to change the cider-test-success-face to something a little less “screamy”.

   ((t (:foreground "green" :background nil)))))

Update <21 Nov 2015>

Made it a global mode, so it’s easy to disable everywhere, and set the initial value to t.

Comment on this.

-1:-- Test-Driven-Development in CIDER and Emacs (Post)--L0--C0--November 16, 2015 12:00 AM

Ivan Kanis: setting up rkhunter

rkhunter is a useful tool to check that your server is not infected with a rootkit.

On debian 8.2 it is missing a dependency so you will have to install it by hand:

apt-get install libwww-perl

Next you will need to run the following command once:

rkhunter --propupd

Edit the file /etc/default/rkhunter and changer the following variable:


This will run rkhunter every day and e-mail you if there is a problem.

Now you need to check that there is no error or warning:

rkhunter --check

As you are fixing the errors or warnings you can narrow down the tests with the following commands:

rkhunter --check --enable filesystem

I left one warning to make sure I got an e-mail the next day.

-1:-- setting up rkhunter (Post Ivan Kanis)--L0--C0--November 15, 2015 12:00 AM

Flickr tag 'emacs': 2015-11-11b A few Emacs community resources -- index card #emacs

sachac posted a photo:

2015-11-11b A few Emacs community resources -- index card #emacs

-1:-- 2015-11-11b A few Emacs community resources -- index card #emacs (Post sachac ( 14, 2015 02:11 AM

Eric James Michael Ritz: GNU Emacs Packages for Python Programming

Lately I’ve been writing a lot of Python code, so tonight I want to talk a little about the GNU Emacs packages I use for Python programming.

Catching Syntax Errors and More

Flycheck is an amazingly useful package for finding programming errors in real-time, and it supports a number of languages right out of the box, including Python. It saves time to be able to see that I’ve made a syntax error or something while writing the code instead of finding out later when I actually run that code.

There are a lot of third-party extensions for Flycheck out there, adding or improving support for different languages. I recommend the following for Python:

  1. flycheck-pyflakes

  2. flycheck-mypy: Recently I wrote about static type-checking in Python using Mypy. This package allows Flycheck to use Mypy to catch type-errors in my code as I write it. It’s not useful for all projects, e.g. anything using Python 2 which Mypy doesn’t support, but when available I’ve found it to be an invaluable tool.

Live Coding

Live-py-mode is a package for Emacs (and Eclipse) which lets you see the results of your Python code in another window as you write it. Effectively it’s like Python running as an "inferior process" while automatically sending all of my code to it, instead of having to use some of Python Mode’s built-in commands like python-shell-send-region.

I recommend watching this video to see just what you can do with live-py-mode.


I’m a fan of Helm, so I use helm-pydoc for quickly looking up documentation on Python modules and libraries. The package can do more than simply bring up documentation though. As you’ll see it also lets me open up the source code for packages or import them into the current buffer using either the import ... or from ... import ... syntax if I want.

Running Tests

The pytest.el package provides a collection of useful commands for running pytest, which I use on most Python projects. If you use Hydra for managing key-bindings then you may find my personal Hydra for pytest.el to be useful:

(define-key python-mode-map (kbd "C-c C-y")
  (defhydra hydra-pytest (:color blue)
    ("a" pytest-all "all")
    ("m" pytest-module "module")
    ("o" pytest-one "one")
    ("d" pytest-directory "directory")))

The Big One

Anaconda is a package which adds a lot of functionality for Python programming. To quote the project page, its features include:

  • context-sensitive code completion
  • jump to definitions
  • find references
  • view documentation
  • virtual environment
  • eldoc mode

If you want a big chunk of additional Python programming features in a single package then you can probably do no better than to use Anaconda. (Edit: I forgot about Elpy, see the comments below.) And if you use Company for auto-completion, which I personally do, then you will also want the Anaconda backend for Company.

-1:-- GNU Emacs Packages for Python Programming (Post ericjmritz)--L0--C0--November 07, 2015 12:25 AM

Ivan Kanis: Fix Intel video acceleration on Xubuntu 14.4

If you get the following error when running mplayer:

Failed to open VDPAU backend cannot open shared object file: No such file or directory

It means that mplayer did not find the video acceleration driver library. This means that your CPU is playing video instead of your video card. This is not the optimal configuration so here are the steps to fix it.

First install the driver:

apt-get install i965-va-driver libvdpau-va-gl1

Next, you will need to have the variable VDPAU_DRIVER set to the GL library name. Edit /etc/environment and add the following line:


Reboot and you should not see the error anymore.

-1:-- Fix Intel video acceleration on Xubuntu 14.4 (Post Ivan Kanis)--L0--C0--November 05, 2015 12:00 AM

Julien Danjou: Gnocchi 1.3.0 release

Finally, Gnocchi 1.3.0 is out. This is our final release, more or less matching the OpenStack 6 months schedule, that concludes the Liberty development cycle.

This release was supposed to be released a few weeks earlier, but our integration test got completely blocked for several days just the week before the OpenStack Mitaka summit.

New website

We build a new dedicated website for Gnocchi at We want to promote Gnocchi outside of the OpenStack bubble, as it a useful timeseries database on its own that can work without the rest of the stack. We'll try to improve the documentation. If you're curious, feel free to check it out and report anything you miss!

The speed bump

Obviously, if it was a bug in Gnocchi that we have hit, it would have been quick to fix. However, we found a nasty bug in Swift caused by the evil monkey-patching of Eventlet (once again) blended with a mixed usage of native threads and Eventlet threads in Swift. Shake all of that, and you got yourself pretty race conditions when using the Keystone middleware authentication.

In the meantime, we disabled Swift multi-threading by using mod_wsgi instead of Eventlet in devstack.

New features

So what's new in this new shiny release? A few interesting things:

  • Metric deletion is now asynchronous. That's not the most used feature in the REST API – weirdly people do not often delete metrics – but it's now way faster and reliable by being asynchronous. Metricd is now in charge of cleaning up things up.

  • Speed improvement. We are now confident to be even more faster than in the latest benchmarks I run (around 1.5-2× faster), which makes Gnocchi really fast with its native storage back-ends. We profiled and optimized Carbonara and the REST API data validation.

  • Improve metricd status report. It now reports the size of the backlog of the whole cluster both in its log and via the REST API. Easy monitoring!

  • Ceph drivers enhancement. We had people testing the Ceph drivers in production, so we made a few changes and fixes to it to make it more solid.

And that's all we did in the last couple of months. We have a lot of things on the roadmap that are pretty exciting, and I'll sure talk about them in the next weeks.

-1:-- Gnocchi 1.3.0 release (Post Julien Danjou)--L0--C0--November 04, 2015 12:58 PM

(or emacs: New feature in Ivy - ivy-occur

I've had an idea with this feature for quite some time, and only recently got to finally implement it. So here it goes: with ivy-occur, bound to C-c C-o while in the minibuffer, you can store almost Ivy any completion session in progress, and you can have as many of them as you like.

Example 1

This is where the command name originates from: swiper + ivy-occur = occur. You can store all of swiper's matching lines into a separate buffer. This doesn't give too much advantage over the good-old occur, the only thing is that you can use ivy-style regexps with wild spaces, and have an interactive preview beforehand. Having said that, it actually sounds pretty good!

Example 2

I press C-S-o, bound to counsel-rhythmbox, and enter u2. After pressing C-c C-o, bound to ivy-occur, the completion session is closed (effectively C-g), and a new buffer *ivy-occur counsel-rhythmbox "u2"* is generated with all songs that match u2.

As a reminder, counsel-rhythmbox has two actions: play (the default) and enqueue. In this new buffer, pressing RET or mouse-1 will call the play action (since it was active at the moment ivy-occur was called) for the current candidate. So I've effectively added a playlist functionality to counsel-rhythmbox through a generic command.

Note that it's very easy to identify a completion session by the command name and input. So I can distinguish e.g. *ivy-occur counsel-rhythmbox "u2"* and *ivy-occur counsel-rhythmbox "скря"*, and quickly select them with ivy-switch-buffer: just input rhy, usually only these two and similar buffers will match.

Example 3

Suppose I want to go through the documentation of all variables that end in -function. Easy:

  1. <f1> v (counsel-describe-variable) with input function$.
  2. C-c C-o (ivy-occur).

I get a new buffer named *ivy-occur cousnel-describe-variable "function$*" with 346 candidates. I can go through them at my own pace, possibly doing other completion stuff in between without disturbing my process. It's also convenient to navigate these buffers with swiper.

Example 4

Let's tweak the previous one. After inputting function$ I can press C-M-a (ivy-read-action) followed by d to select the definition action. Then again C-c C-o (ivy-occur). Pressing RET in the resulting buffer will take me to that variable's definition, instead of describing it as before. A similar thing could be done for counsel-rhythmbox to get the enqueue action in *ivy-occur* buffer, instead of play.

Example 5

This is an improvement to my workflow for quickly looking at a new package's features. This can be done with oge (lispy-goto-elisp-commands) from lispy, which scans the source for (interactive) tags and allows to jump to their definition.

It's a pretty cool optimization, compared to looking at all tags. For example, projectile currently has 375 top-level tags (functions and variables). But with lispy-goto-elisp-commands I only get 48 tags. And now I can have them in a convenient permanent list too.


Alternatively, if projectile is already loaded, I can use counsel-M-x with input projectile-, followed by C-M-a d to select the definition action, followed by C-c C-o.

As a reminder of how it works, counsel-M-x is defined with a single action that calls the selected command. But then, I've also added this statement at top-level of counsel.el:

 '(("d" counsel--find-symbol "definition")))

This means that you can add as many actions as you like to ivy-read commands. And of course customize the binding and the hint, which are in this case d and definition respectively.


Unfortunately, since the *ivy-occur* buffer needs to know the action to execute, it only works for commands that explicitly pass :action to ivy-read. For instance, it won't work for package-install with ivy-mode on: the buffer will be properly generated, but pressing RET won't install a package.

Fortunately, it's not hard to write a version that works:

(defun counsel-package-install ()
  (ivy-read "Install package: "
            (delq nil
                  (mapcar (lambda (elt)
                            (unless (package-installed-p (car elt))
                              (symbol-name (car elt))))
            :action (lambda (x)
                      (package-install (intern x)))
            :caller 'counsel-package-install))

Here's a buffer with a list of packages matching "ga"; pressing RET will install the selected package:


Small note on key bindings

In the initial post, I wanted to bind ivy-occur to C-c o instead of C-c C-o. But I was reminded that C-c LETTER are reserved. I still think it's a better binding. If you agree, you can add it to your config:

(define-key ivy-minibuffer-map (kbd "C-c o") 'ivy-occur)

Additionally, ivy-occur is also available on C-o u, through the C-o hydra.


I think ivy-occur is a very powerful command that shouldn't be overlooked. Just as ivy-resume implements a sort of DEL or C-u key for completion, ivy-occur implements a convenient way to switch and store the completion context, a sort of C-x o or C-x b for completion.

-1:-- New feature in Ivy - ivy-occur (Post)--L0--C0--November 03, 2015 11:00 PM

Flickr tag 'emacs': 2015-11-02c Emacs News notes -- index card #emacs #kaizen

sachac posted a photo:

2015-11-02c Emacs News notes -- index card #emacs #kaizen

-1:-- 2015-11-02c Emacs News notes -- index card #emacs #kaizen (Post sachac ( 03, 2015 04:02 AM

Rubén Berenguel: SyncTeX and pdf-view-mode for emacs

Or, destiny is cruel
Back in the days of yore, when I was switching between my Windows machine and a Linux machine, I remember having SyncTeX active in my Windows machine. It was a wonderful experience: SyncTeX lets you click anywhere on a generated file from LaTeX and gets back to your editor, to the place generating where you clicked. This was extremely useful, specially later on when you need to adjust many formulas to fit and you need a bit of back-and-forth-ing.

Then I got a Mac, and since Preview is so handy I slowly forgot about SyncTeX. Time went on, and I merrily kept on editing LaTeX files as usual. I even managed to deliver my PhD dissertation a couple weeks ago, the formal speech will be in a month or two (come at your own risk). AucTeX’s preview saved most of the days, so I slowly even forgot SyncTeX existed. Shame on me indeed.

The other day I got an annotated PDF from one of my advisors, and I just couldn’t open the annotations. I tried all programs I had for Mac, and no luck: annotations weren’t showing, just saw the icons. Surveying for some command-line tool to extract annotations (just in case) I found pdf-tools, a replacement for DocView based on Poppler. It had the awesome ability of actually displaying annotations, with it it was pretty clear the annotations were broken in that PDF. I got a new set of PDFs from my advisor with the annotations in place, though. While waiting for it to arrive…

I saw SyncTeX was an option of pdf-tools. I had been using that, hadn’t I? So, I activated SyncTeX in AucTeX (it is TeX-source-correlate-method, see here) and indeed: I could have two frames, one with the actual LaTeX sources and the other with a PDF, and go from one to the other. Even hyperreferences in PDF work! See (well, click on the full-screen mode to see it larger or you won't see anything)!

Getting pdf-tools to work wasn’t incredibly tricky (given the hoops you need for some packages, sometimes). Just

brew install pdf-tools

and after reading

brew info pdf-tools

I was told to run

  emacs -Q --batch --eval "(package-install-file \"/usr/local/Cellar/pdf-tools/0.60/pdf-tools-0.60.tar\")"

and this does the trick (well, change emacs for your actual emacs, which likely is /Applications/ You’ll also need to add to your .emacs file (or temporarily in your *scratch* buffer)

(setenv "PKG_CONFIG_PATH" (concat "/usr/local/Cellar/zlib/1.2.8/lib/pkgconfig" ":" "/usr/local/lib/pkgconfig:/opt/X11/lib/pkgconfig"))

(getenv "PKG_CONFIG_PATH")

and run


as advised in the package’s README. And that's it, open a PDF and activate pdf-view-mode to check everything is in place. Well worth it!
-1:-- SyncTeX and pdf-view-mode for emacs (Post Rubén Berenguel ( 02, 2015 10:34 PM

Jorgen Schäfer: Elpy 1.10.0 released

I just released version 1.10.0 of Elpy, the Emacs Python Development Environment. This is a feature release.

Elpy is an Emacs package to bring powerful Python editing to Emacs. It combines a number of other packages, both written in Emacs Lisp as well as Python.

Quick Installation

Evaluate this:

(require 'package)
(add-to-list 'package-archives
'("elpy" .

Then run M-x package-install RET elpy RET.

Finally, run the following (and add them to your .emacs):


Changes in 1.10.0

  • Marking the current indentation level at the top level will now mark the whole buffer.
  • The region will be normalized before re-indenting it, making the behavior more predictable for partially marked lines.
  • Using autopep8 on the whole buffer will now keep point (roughly) at the same location as it was.
  • The autopep8 code now also uses the same configuration options as the command line tool.
  • Malformed JSON data from the backend is now handled better.
  • RPC processes are restarted when the current virtualenv changes.
  • Python 3.5 is now officially supported.
  • Flymake will now not be enabled in buffers without file name, where it can't run anyhow, or when the checker program does not exist in the first place.
  • It is now possible to ask Elpy not to remove the mode line lighter of company mode, which can be useful if you use company in other modes.
  • Test discovery now also allows file names without "test" in them to be tested. Classes and methods still require the substring, though.
  • Spurious equals signs at the end of completions from Jedi will now be ignored.
  • Various other bug fixes.

Thanks to ChillarAnand and Georg Brandl for their contributions!

-1:-- Elpy 1.10.0 released (Post Jorgen Schäfer ( 01, 2015 12:06 PM