Irreal: Three Groups of Editors

I came across this tweet

that reminded me of my slightly less controversial post on the same subject. That post was from 2 and a half years ago so it's worth considering how things have changed.

The most obvious change is that TextMate is no longer a player. It's apparently still available and the source code of the new version development has moved to GitHub but one hardly hears about it anymore. In terms of my original post, that leaves Emacs and Vim as the editors of choice for experienced developers. The battle between them for the top spot proceeds apace, of course, with partisans on both sides as adamant as ever.

There are a couple of new editors that are getting a lot of buzz: Sublime Text and Atom. Emacs users haven't been much impressed and I assume the same is true of Vim users as well. Emacs and Vim are like black holes. Once you start using them you never leave1. It may be that in the long term these editors will gain traction but in the mean time, I think it's still true that the most experienced engineers are using Emacs or Vim.

There are lots of decent editors, each with their own champions, and plenty of experienced and talented developers who use them. Still, as a general statement I think it's still true that the most experienced and talented engineers tend to prefer Emacs or Vim, and that to a first order approximation their use signals someone who is experienced and cares about optimizing their work flow.

On the other hand, as far as uptake goes, we have this.



Although there is movement between the two editors. It's just that there's no escape from the twin system of Emacs and Vim.

-1:-- Three Groups of Editors (Post jcs)--L0--C0--September 16, 2014 11:46 AM

emacspeak: Emacspeak At Twenty: Looking Back, Looking Forward

Emacspeak At Twenty: Looking Back, Looking Forward

1 Introduction

One afternoon in the third week of September 1994, I started writing myself a small Emacs extension using Lisp Advice to make Emacs speak to me so I could use a Linux laptop. As Emacspeak turns twenty, this article is both a quick look back over the twenty years of lessons learned, as well as a glimpse into what might be possible as we evolve to a world of connected, ubiquitous computing. This article draws on Learning To Program In 10 Years by Peter Norvig for some of its inspiration.

2 Using UNIX With Speech Output — 1994

As a graduate student at Cornell, I accessed my Unix workstation (SunOS) from an Intel 486 PC running IBM Screen-Reader. There was no means of directly using a UNIX box at the time; after graduating, I continued doing the same for about six months at Digital Research in Cambridge — the only difference being that my desktop workstation was now a DEC-Alpha. Throughout this time, Emacs was my environment of choice for everything from software development and Internet access to writing documents.

In fall of 1994, I wanted to start using a laptop running Linux; a colleague (Dave Wecker) was retiring his 386mhz laptop that already had Linux on it and I decided to inherit it. But there was only one problem — until then I had always accessed a UNIX machine from a secondary PC running a screen-reader — something that would clearly make no sense with a laptop!

Another colleague, Win Treese, had pointed out the interesting possibilities presented by package advice in Emacs 19.23 — a few weeks earlier, he had sent around a small snippet of code that magically modified Emacs' version-control primitive to first create an RCS directory if none existed before adding a file to version control. When I speculated about using the Linux laptop, Dave remarked — you live in Emacs anyway — why dont you just make it talk!

Connecting the dots, I decided to write myself a tool that augmented Emacs' default behavior to speak — within about 4 hours, version 0.01 of Emacspeak was up and running.

3 Key Enabler — Emacs And Lisp Advice

It took me a couple of weeks to fully recognize the potential of what I had built with Emacs Lisp Advice. Until then, I had used screen-readers to listen to the contents of the visual display — but Lisp Advice let me do a lot more — it enabled Emacspeak to generate highly context-specific spoken feedback, augmented by a set of auditory icons. I later formalized this design under the name speech-enabled applications. For a detailed overview of the architecture of Emacspeak, see the chapter on Emacspeak in the book Beautiful Code from O'Reilly.

4 Key Component — Text To Speech (TTS)

Emacspeak is a speech-subsystem for Emacs; it depends on an external Text-To-Speech (TTS) engine to produce speech. In 1994, Digital Equipment released what would turn out to be the last in the line of hardware DECTalk synthesizers, the DECTalk Express. This was essentially an Intel 386with 1mb of flash memory that ran a version of the DECTalk TTS software — to date, it still remains my favorite Text-To-Speech engine. At the time, I also had a software version of the same engine running on my DEC-Alpha workstation; the desire to use either a software or hardware solution to produce speech output defined the Emacspeak speech-server architecture.

I went to IBM Research in 1999; this coincided with IBM releasing a version of the Eloquennce TTS engine on Linux under the name ViaVoice Outloud. My colleague Jeffrey Sorenson implemented an early version of the Emacspeak speech-server for this engine using the OSS API; I later updated it to use the ALSA library while on a flight back to SFO from Boston in 2001. That is still the TTS engine that is speaking as I type this article on my laptop.

20 years on, TTS continues to be the weakest link on Linux; the best available solution in terms of quality continues to be the Linux port of Eloquence TTS available from Voxin in Europe for a small price. Looking back across 20 years, the state of TTS on Linux in particular and across all platforms in general continues to be a disappointment; most of today's newer TTS engines are geared toward mainstream use-cases where naturalness of the voice tends to supersede intelligibility at higher speech-rates. Ironically, modern TTS engines also give applications far less control over the generated output — as a case in point, I implemented Audio System For Technical Readings (AsTeR) in 1994 using the DECTalk; 20 years later, we implemented MathML support in ChromeVox using Google TTS. In 2013, it turned out to be difficult or impossible to implement the type of audio renderings that were possible with the admittedly less-natural sounding DECTalk!

5 Emacspeak And Software Development

Version 0.01 of Emacspeak was written using IBM Screen-Reader on a PC with a terminal emulator accessing a UNIX workstation. But in about 2 weeks, Emacspeak was already a better environment for developing Emacspeak in particular and software development in general. Here are a few highlights in 1994 that made Emacspeak a good software development environment, present-day users of Emacspeak will see that that was just scratching the surface.

  • Audio formatting using voice-lock to provide aural syntax highlighting.
  • Succinct auditory icons to provide efficient feedback.
  • Emacs' ability to navigate code structurally —

as opposed to moving around by plain-text units such as characters, lines and words. S-Expressions are a major win!

  • Emacs' ability to specialize behavior based on major and minor modes.
  • Ability to browse program code using tags, and getting fluent spoken feedback.
  • Completion everywhere.
  • Everything is searchable — this is a huge win when you cannot see the screen.
  • Interactive spell-checking using ISpell with continuous spoken feedback augmented by aural highlights.
  • Running code compilation and being able to jump to errors with spoken feedback.
  • Ability to move through diff chunks when working with source code and source control systems; refined diffs as provided by the ediff package when speech-enabled is a major productivity win.
  • Ability to easily move between email, document authoring and programming — though this may appear trivial, it continues to be one of Emacs' biggest wins.

Long-term Emacs users will recognize all of the above as being among the reasons why they do most things inside Emacs — there is little that is Emacspeak specific in the above list — except that Emacspeak was able to provide fluent, well-integrated contextual feedback for all of these tasks. And that was a game-changer given what I had had before Emacspeak. As a case in point, I did not dare program in Python before I speech-enabled Emacs' Python-Mode; the fact that white space is significant in Python made it difficult to program using a plain screen-reader that was unaware of the semantics of the underlying content being accessed.

5.1 Programming Defensively

As an aside, note that all of Emacspeak has been developed over the last 20 years with Emacspeak being the only adaptive technology on my system. This has led to some interesting design consequences, primary among them being a strong education in programming defensively. Here are some other key features of the Emacspeak code-base:

  1. The code-base is extremely bushy rather than deeply hierarchical — this means that when a module breaks, it does not affect the rest of the system.
  2. Separation of concerns with respect to the various layers, a tightly knit core speech library interfaces with any one of many speech servers running as an external process.
  3. Audio formatting is abstracted by using the formalism defined in Aural CSS.
  4. Emacspeak integrates with Emacs' user interface conventions by taking over a single prefix key C-e with all Emacspeak commands accessed through that single keymap. This helps embedding Emacspeak functionality into a large variety of third party modules without any loss of functionality.

6 Emacspeak And Authoring Documents

In 1994, my preferred environment for authoring all documents was LaTeX using the Auctex package. Later I started writing either LaTeX or HTML using the appropriate support modes; today I use org-mode to do most of my content authoring. Personally, I have never been a fan of What You See Is What You Get (WYSIWYG) authoring tools — in my experience that places an undue burden on the author by drawing attention away from the content to focus on the final appearance. An added benefit of creating content in Emacs in the form of light-weight markup is that the content is long-lived — I can still usefully process and re-use things I have written 25 years ago.

Emacs, with Emacspeak providing audio formatting and context-specific feedback remains my environment of choice for writing all forms of content ranging from simple email messages to polished documents for print publishing. And it is worth repeating that I never need to focus on what the content is going to look like — that job is best left to the computer.

As an example of producing high-fidelity visual content, see this write-up on Polyhedral Geometry that I published in 2000; all of the content, including the drawings were created by me using Emacs.

7 Emacspeak And The Early Days Of The Web

Right around the time that I was writing version 0.01 of emacspeak, a far more significant software movement was under way — the World Wide Web was moving from the realms of academia to the mainstream world with the launch of NCSA Mosaic — and in late 1994 by the first commercial Web browser in Netscape Navigator. Emacs had always enabled integrated access to FTP archives via package ange-ftp; in late 1993, William Perry released Emacs-W3, a Web browser for Emacs written entirely in Emacs Lisp. W3 was one of the first large packages to be speech-enabled by Emacspeak — later it was the browser on which I implemented the first draft of the Aural CSS specification. Emacs-W3 enabled many early innovations in the context of providing non-visual access to Web content, including audio formatting and structured content navigation; in summer of 1995, Dave Raggett and I outlined a few extensions to HTML Forms, including the label element as a means of associating metadata with interactive form controls in HTML, and many of these ideas were prototyped in Emacs-W3 at the time. Over the years, Emacs-W3 fell behind the times — especially as the Web moved away from cleanly structured HTML to a massive soup of unmatched tags. This made parsing and error-correcting badly-formed HTML markup expensive to do in Emacs-Lisp — and performance suffered. To add to this, mainstream users moved away because Emacs' rendering engine at the time was not rich enough to provide the type of visual renderings that users had come to expect. The advent of DHTML, and JavaScript based Web Applications finally killed off Emacs-W3 as far as most Emacs users were concerned.

But Emacs-W3 went through a revival on the emacspeak audio desktop in late 1999 with the arrival of XSLT, and Daniel Veillard's excellent implementation via the libxml2 and libxslt packages. With these in hand, Emacspeak was able to hand-off the bulk of HTML error correction to the xsltproc tool. The lack of visual fidelity didn't matter much for an eyes-free environment; so Emacs-W3 continued to be a useful tool for consuming large amounts of Web content that did not require JavaScript support.

During the last 24 months, libxml2 has been built into Emacs; this means that you can now parse arbitrary HTML as found in the wild without incurring a performance hit. This functionality was leveraged first by package shr (Simple HTML Renderer) within the gnus package for rendering HTML email. Later, the author of gnus and shr created a new light-weight HTML viewer called eww that is now part of Emacs 24. With improved support for variable pitch fonts and image embedding, Emacs is once again able to provide visual renderings for a large proportion of text-heavy Web content where it becomes useful for mainstream Emacs users to view at least some Web content within Emacs; during the last year, I have added support within emacspeak to extend package eww with support for DOM filtering and quick content navigation.

8 Audio Formatting — Generalizing Aural CSS

A key idea in Audio System For Technical Readings (AsTeR) was the use of various voice properties in combination with non-speech auditory icons to create rich aural renderings. When I implemented Emacspeak, I brought over the notion of audio formatting to all buffers in Emacs by creating a voice-lock module that paralleled Emacs' font-lock module. The visual medium is far richer in terms of available fonts and colors as compared to voice parameters available on TTS engines — consequently, it did not make sense to directly map Emacs' face properties to voice parameters. To aid in projecting visual formatting onto auditory space, I created property personality analogous to Emacs' face property that could be applied to content displayed in Emacs; module voice-lock applied that property appropriately, and the Emacspeak core handled the details of mapping personality values to the underlying TTS engine.

The values used in property personality were abstract, i.e., they were independent of any given speech engine. Later in the fall of 1995, I re-expressed these set of abstract voice properties in terms of Aural CSS; the work was published as a first draft toward the end of 1995, and implemented in Emacs-W3 in early 1996. Aural CSS was an appendix in the CSS-1.0 specification; later, it graduated to being its own module within CSS-2.0.

Later in 1996, all of Emacs' voice-lock functionality was re-implemented in terms of Aural CSS; the implementation has stood the test of time in that as I added support for more TTS engines, I was able to implement engine-specific mappings of Aural-CSS values. This meant that the rest of Emacspeak could define various types of voices for use in specific contexts without having to worry about individual TTS engines. Conceptually, property personality can be thought of as holding an aural display list — various parts of the system can annotate pieces of text with relevant properties that finally get rendered in the aggregate. This model also works well with the notion of Emacs overlays where a moving overlay is used to temporarily highlight text that has other context-specific properties applied to it.

Audio formatting as implemented in Emacspeak is extremely effective when working with all types of content ranging from richly structured mark-up documents (LaTeX, org-mode) and formatted Web pages to program source code. Perceptually, switching to audio formatted output feels like switching from a black-and-white monitor to a rich color display. Today, Emacspeak's audio formatted output is the only way I can correctly write else if vs elsif in various programming languages!

9 Conversational Gestures For The Audio Desktop

By 1996, Emacspeak was the only piece of adaptive technology I used; in fall of 1995, I had moved to Adobe Systems from DEC Research to focus on enhancing the Portable Document Format (PDF) to make PDF content repurposable. Between 1996 and 1998, I was primarily focused on electronic document formats — I took this opportunity to step back and evaluate what I had built as an auditory interface within Emacspeak. This retrospect proved extremely useful in gaining a sense of perspective and led to formalizing the high-level concept of Conversational Gestures and structured browsing/searching as a means of thinking about user interfaces.

By now, Emacspeak was a complete environment — I formalized what it provided under the moniker Complete Audio Desktop. The fully integrated user experience allowed me to move forward with respect to defining interaction models that were highly optimized to eyes-free interaction — as an example, see how Emacspeak interfaces with modes like dired (Directory Editor) for browsing and manipulating the filesystem, or proced (Process Editor) for browsing and manipulating running processes. Emacs' integration with ispell for spell checking, as well as its various completion facilities ranging from minibuffer completion to other forms of dynamic completion while typing text provided more opportunities for creating innovative forms of eyes-free interaction. With respect to what had gone before (and is still par for the course as far as traditional screen-readers are concerned), these types of highly dynamic interfaces present a challenge. For example, consider handling a completion interface using a screen-reader that is speaking the visual display. There is a significant challenge in deciding what to speak e.g., when presented with a list of completions, the currently typed text, and the default completion, which of these should you speak, and in what order? The problem gets harder when you consider that the underlying semantics of these items is generally not available from examining the visual presentation in a consistent manner. By having direct access to the underlying information being presented, Emacspeak had a leg up with respect to addressing the higher-level question — when you do have access to this information, how do you present it effectively in an eyes-free environment? For this and many other cases of dynamic interaction, a combination of audio formatting, auditory icons, and the ability to synthesize succinct messages from a combination of information items — rather than having to forcibly speak each item as it is rendered visually provided for highly efficient eyes-free interaction.

This was also when I stepped back to build out Emacspeak's table browsing facilities — see the online Emacspeak documentation for details on Emacspeak's table browsing functionality which continues to remain one of the richest collection of end-user affordances for working with two-dimensional data.

9.1 Speech-Enabling Interactive Games

So in 1997, I went the next step in asking — given access to the underlying infromation, is it possible to build effective eyes-free interaction to highly interactive tasks? I picked Tetris as a means of exploring this space, the result was an Emacspeak extension to speech-enable module tetris.el. The details of what was learned were published as a paper in Assets 98, and expanded as a chapter on Conversational Gestures in my book on Auditory Interfaces; that book was in a sense a culmination of stepping back and gaining a sense of perspective of what I had build during this period. The work on Conversational Gestures also helped in formalizing the abstract user interface layer that formed part of the XForms work at the W3C.

Speech-enabling games for effective eyes-free interaction has proven highly educational. Interactive games are typically built to challenge the user, and if the eyes-free interface is inefficient, you just wont play the game — contrast this with a task that you must perform, where you're likely to make do with a sub-optimal interface. Over the years, Emacspeak has come to include eyes-free interfaces to several games including Tetris, Sudoku, and of late the popular 2048 game. Each of these have in turn contributed to enhancing the interaction model in Emacspeak, and those innovations typically make their way to the rest of the environment.

10 Accessing Media Streams

Streaming real-time audio on the Internet became a reality with the advent of RealAudio in 1995; soon there were a large number of media streams available on the Internet ranging from music streams to live radio stations. But there was an interesting twist — for the most part, all of these media streams expected one to look at the screen, even though the primary content was purely audio (streaming video hadn't arrived yet!). Starting in 1996, Emacspeak started including a variety of eyes-free front-ends for accessing media streams. Initially, this was achieved by building a wrapper around trplayer — a headless version of RealPlayer; later I built Emacspeak module emacspeak-m-player for interfacing with package mplayer. A key aspect of streaming media integration in emacspeak is that one can launch and control streams without ever switching away from one's primary task; thus, you can continue to type email or edit code while seamlessly launching and controlling media streams. Over the years, Emacspeak has come to integrate with Emacs packages like emms as well as providing wrappers for mplayer and alsaplayer — collectively, these let you efficiently launch all types of media streams, including streaming video, without having to explicitly switch context.

In the mid-90's, Emacspeak started including a directory of media links to some of the more popular radio stations — primarily as a means of helping users getting started — Emacs' ability to rapidly complete directory and file-names turned out to be the most effective means of quickly launching everything from streaming radio stations to audio books. And even better — as the Emacs community develops better and smarter ways of navigating the filesystem using completions, e.g., package ido, these types of actions become even more efficient!

11 EBooks— Ubiquitous Access To Books

AsTeR — was motivated by the increasing availability of technical material as online electronic documents. While AsTeR processed the TeX family of markup languages, more general ebooks came in a wide range of formats, ranging from plain text generated from various underlying file formats to structured EBooks, with Project Gutenberg leading the way. During the mid-90's, I had access to a wide range of electronic materials from sources such as O'Reilly Publishing and various electronic journals — The Perl Journal (TPJ) is one that I still remember fondly.

Emacspeak provided fairly light-weight but efficient access to all of the electronic books I had on my local disk — Emacs' strengths with respect to browsing textual documents meant that I needed to build little that was specific to Emacspeak. The late 90's saw the arival of Daisy as an XML-based format for accessible electronic books. The last decade has seen the rapid convergence to epub as a distribution format of choice for electronic books. Emacspeak provides interaction modes that make organizing, searching and reading these materials on the Emacspeak Audio Desktop a pleasant experience. Emacspeak also provides an OCR-Mode — this enables one to call out to an external OCR program and read the content efficiently.

The somewhat informal process used by publishers like O'Reilly to make technical material available to users with print impairments was later formalized by BookShare — today, qualified users can obtain a large number of books and periodicals initially as Daisy-3 and increasingly as EPub. BookShare provides a RESTful API for searching and downloading books; Emacspeak module emacspeak-bookshare implements this API to create a client for browsing the BookShare library, downloading and organizing books locally, and an integrated ebook reading mode to round off the experience.

A useful complement to this suite of tools is the Calibre package for organizing ones ebook collection; Emacspeak now implements an EPub Interaction mode that leverages Calibre (actually sqlite3) to search and browse books, along with an integrated EPub mode for reading books.

12 Leveraging Computational Tools — From SQL And R To IPython Notebooks

The ability to invoke external processes and interface with them via a simple read-eval-loop (REPL) is perhaps one of Emacs' strongest extension points. This means that a wide variety of computational tools become immediately available for embedding within the Emacs environment — a facility that has been widely exploited by the Emacs community. Over the years, Emacspeak has leveraged many of these facilities to provide a well-integrated auditory interface.

Starting from a tight code, eval, test form of iterative programming as encouraged by Lisp. Applied to languages like Python and Ruby to explorative computational tools such as R for data analysis and SQL for database interaction, the Emacspeak Audio Desktop has come to encompass a collection of rich computational tools that provide an efficient eyes-free experience.

In this context, module ein — Emacs IPython Notebooks — provides another excellent example of an Emacs tool that helps interface seamlessly with others in the technical domain. IPython Notebooks provide an easy means of reaching a large audience when publishing technical material with interactive computational content; module ein brings the power and convenience of Emacs ' editting facilities when developing the content. Speech-enabling package ein is a major win since editting program source code in an eyes-free environment is far smoother in Emacs than in a browser-based editor.

13 Social Web — EMail, Instant Messaging, Blogging And Tweeting Using Open Protocols

The ability to process large amounts of email and electronic news has always been a feature of Emacs. I started using package vm for email in 1990, along with gnus for Usenet access many years before developing Emacspeak. So these were the first major packages that Emacspeak speech-enabled. Being able to access the underlying data structures used to visually render email messages and Usenet articles enabled Emacspeak to produce rich, succinct auditory output — this vastly increased my ability to consume and organize large amounts of information. Toward the turn of the century, instant messaging arrived in the mainstream — package tnt provided an Emacs implementation of a chat client that could communicate with users on the then popular AOL Instant Messenger platform. At the time, I worked at IBM Research, and inspired by package tnt, I created an Emacs client called ChatterBox using the Lotus Sametime API — this enabled me to communicate with colleagues at work from the comfort of Emacs. Packages like vm, gnus, tnt and ChatterBox provide an interesting example of how availability of a clean underlying API to a specific service or content stream can encourage the creation of efficient (and different) user interfaces. The touchstone of such successful implementations is a simple test — can the user of a specific interface tell if the person whom he is communicating with is also using the same interface? In each of the examples enumerated above, a user at one end of the communication chain cannot tell, and in fact shouldn't be able to tell what client the user at the other end is using. Contrast this with closed services that have an inherent lock-in model e.g., proprietary word processors that use undocumented serialization formats — for a fun read, see this write-up on Universe Of Fancy Colored Paper.

Today, my personal choice for instant messaging is the open Jabber platform. I connect to Jabber via Emacs package emacs-jabber and with Emacspeak providing a light-weight wrapper for generating the eyes-free interface, I can communicate seamlessly with colleagues and friends around the world.

As the Web evolved to encompass ever-increasing swathes of communication functionality that had already been available on the Internet, we saw the world move from Usenet groups to Blogs — I remember initially dismissing the blogging phenomenon as just a re-invention of Usenet in the early days. However, mainstream users flocked to Blogging, and I later realized that blogging as a publishing platform brought along interesting features that made communicating and publishing information much easier. In 2005, I joined Google; during the winter holidays that year, I implemented a light-weight client for Blogger that became the start of Emacs package g-client — this package provides Emacs wrappers for Google services that provide a RESTful API.

14 The RESTful Web — Web Wizards And URL Templates For Faster Access

Today, the Web, based on URLs and HTTP-style protocols is widely recognized as a platform in its own right. This platform emerged over time — to me, Web APIs arrived in the late 90's when I observed the following with respect to my own behavior on many popular sites:

  1. I opened a Web page that took a while to load (remember, I was still using Emacs-W3),
  2. I then searched through the page to find a form-field that I filled out, e.g., start and end destinations on Yahoo Maps,
  3. I hit submit, and once again waited for a heavy-weight HTML page to load,
  4. And finally, I hunted through the rendered content to find what I was looking for.

This pattern repeated across a wide-range of interactive Web sites ranging from AltaVista for search (this was pre-Google), Yahoo Maps for directions, and Amazon for product searches to name but a few. So I decided to automate away the pain by creating Emacspeak module emacspeak-websearch that did the following:

  1. Prompt via the minibuffer for the requisite fields,
  2. Consed up an HTTP GET URL,
  3. Retrieved this URL,
  4. And filtered out the specific portion of the HTML DOM that held the generated response.

Notice that the above implementation hard-wires the CGI parameter names used by a given Web application into the code implemented in module emacspeak-websearch. REST as a design pattern had not yet been recognized, leave alone formalized, and module emacspeak-websearch was initially decryed as being fragile.

However, over time, the CGI parameter names remained fixed — the only things that have required updating in the Emacspeak code-base are the content filtering rules that extract the response — for popular services, this has averaged about one to two times a year.

I later codified these filtering rules in terms of XPath, and also integrated XSLT-based pre-processing of incoming HTML content before it got handed off to Emacs-W3 — and yes, Emacs/Advice once again came in handy with respect to injecting XSLT pre-processing into Emacs-W3!

Later, in early 2000, I created companion module emacspeak-url-templates — partially inspired by Emacs' webjump module. URL templates in Emacspeak leveraged the recognized REST interaction pattern to provide a large collection of Web widgets that could be quickly invoked to provide rapid access to the right pieces of information on the Web.

The final icing on the cake was the arrival of RSS and Atom feeds and the consequent deep-linking into content-rich sites — this meant that Emacspeak could provide audio renderings of useful content without having to deal with complex visual navigation! While Google Reader existed, Emacspeak provided a light-weight greader client for managing ones feed subscriptions; with the demise of Google Reader, I implemented module emacspeak-feeds for organizing feeds on the Emacspeak desktop. A companion package emacspeak-webspace implements additional goodies including a continuously updating ticker of headlines taken from the user's collection of subscribed feeds.

15 Mashing It Up — Leveraging Evolving Web APIs

The next step in this evolution came with the arrival of richer Web APIs — especially ones that defined a clean client/server separation. In this respect, the world of Web APIs is a somewhat mixed bag in that many Web sites equate a Web API with a JS-based API that can be exclusively invoked from within a Web-Browser run-time. The issue with that type of API binding is that the only runtime that is supported is a full-blown Web browser; but the arrival of native mobile apps has actually proven a net positive in encouraging sites to create a cleaner separation. Emacspeak has leveraged these APIs to create Emacspeak front-ends to many useful services, here are a few:

  1. Minibuffer completion for Google Search using Google Suggest to provide completions.
  2. Librivox for browsing and playing free audio books.
  3. NPR for browsing and playing NPR archived programs.
  4. BBC for playing a wide variety of streaming content available from the BBC.
  5. A Google Maps front-end that provides instantaneous access to directions and Places search.
  6. Access to Twitter via package twittering-mode.

And a lot more than will fit this margin! This is an example of generalizing the concept of a mashup as seen on the Web with respect to creating hybrid applications by bringing together a collection of different Web APIs. Another way to think of such separation is to view an application as a head and a body — where the head is a specific user interface, with the body implementing the application logic. A cleanly defined separation between the head and body allows one to attach different user interfaces i.e., heads to the given body without any loss of functionality, or the need to re-implement the entire application. Modern platforms like Android enable such separation via an Intent mechanism. The Web platform as originally defined around URLs is actually well-suited to this type of separation — though the full potential of this design pattern remains to be fully realized given today's tight association of the Web to the Web Browser.

16 Conclusion

In 1996, I wrote an article entitled User Interface — A Means To An End pointing out that the size and shape of computers were determined by the keyboard and display. This is even more true in today's world of tablets, phablets and large-sized phones — with the only difference that the keyboard has been replaced by a touch screen. The next generation in the evolution of personal devices is that they will become truly personal by being wearables — this once again forces a separation of the user interface peripherals from the underlying compute engine. Imagine a variety of wearables that collectively connect to ones cell phone, which itself connects to the cloud for all its computational and information needs. Such an environment is rich in possibilities for creating a wide variety of user experiences to a single underlying body of information; Eyes-Free interfaces as pioneered by systems like Emacspeak will come to play an increasingly vital role alongside visual interaction when this comes to pass.

–T.V. Raman, San Jose, CA, September 12, 2014

17 References

-1:-- Emacspeak At Twenty: Looking Back, Looking Forward (Post T. V. Raman ( 15, 2014 09:11 PM

Irreal: The Real Purpose of Operating Systems

An amusing anecdote from Perry Metzger's The Editor of a Lifetime talk.

-1:-- The Real Purpose of Operating Systems (Post jcs)--L0--C0--September 15, 2014 11:43 AM

Julien Danjou: Python bad practice, a concrete case

A lot of people read up on good Python practice, and there's plenty of information about that on the Internet. Many tips are included in the book I wrote this year, The Hacker's Guide to Python. Today I'd like to show a concrete case of code that I don't consider being the state of the art.

In my last article where I talked about my new project Gnocchi, I wrote about how I tested, hacked and then ditched whisper out. Here I'm going to explain part of my thought process and a few things that raised my eyebrows when hacking this code.

Before I start, please don't get the spirit of this article wrong. It's in no way a personal attack to the authors and contributors (who I don't know). Furthermore, whisper is a piece of code that is in production in thousands of installation, storing metrics for years. While I can argue that I consider the code not to be following best practice, it definitely works well enough and is worthy to a lot of people.


The first thing that I noticed when trying to hack on whisper, is the lack of test. There's only one file containing tests, named, and the coverage it provides is pretty low. One can check that using the coverage tool.

$ coverage run
Ran 11 tests in 0.014s
$ coverage report
Name Stmts Miss Cover
test_whisper 134 4 97%
whisper 584 227 61%
TOTAL 718 231 67%

While one would think that 61% is "not so bad", taking a quick peak at the actual test code shows that the tests are incomplete. Why I mean by incomplete is that they for example use the library to store values into a database, but they never check if the results can be fetched and if the fetched results are accurate. Here's a good reason one should never blindly trust the test cover percentage as a quality metric.

When I tried to modify whisper, as the tests do not check the entire cycle of the values fed into the database, I ended up doing wrong changes but had the tests still pass.

No PEP 8, no Python 3

The code doesn't respect PEP 8 . A run of flake8 + hacking shows 732 errors… While it does not impact the code itself, it's more painful to hack on it than it is on most Python projects.

The hacking tool also shows that the code is not Python 3 ready as there is usage of Python 2 only syntax.

A good way to fix that would be to set up tox and adds a few targets for PEP 8 checks and Python 3 tests. Even if the test suite is not complete, starting by having flake8 run without errors and the few unit tests working with Python 3 should put the project in a better light.

Not using idiomatic Python

A lot of the code could be simplified by using idiomatic Python. Let's take a simple example:

def fetch(path,fromTime,untilTime=None,now=None):
fh = None
fh = open(path,'rb')
return file_fetch(fh, fromTime, untilTime, now)
if fh:

That piece of code could be easily rewritten as:

def fetch(path,fromTime,untilTime=None,now=None):
with open(path, 'rb') as fh:
return file_fetch(fh, fromTime, untilTime, now)

This way, the function looks actually so simple that one can even wonder why it should exists – but why not.

Usage of loops could also be made more Pythonic:

for i,archive in enumerate(archiveList):
if i == len(archiveList) - 1:

could be actually:

for i, archive in enumerate(itertools.islice(archiveList, len(archiveList) - 1):

That reduce the code size and makes it easier to read through the code.

Wrong abstraction level

Also, one thing that I noticed in whisper, is that it abstracts its features at the wrong level.

Take the create() function, it's pretty obvious:

def create(path,archiveList,xFilesFactor=None,aggregationMethod=None,sparse=False,useFallocate=False):
# Set default params
if xFilesFactor is None:
xFilesFactor = 0.5
if aggregationMethod is None:
aggregationMethod = 'average'
#Validate archive configurations...
#Looks good, now we create the file and write the header
if os.path.exists(path):
raise InvalidConfiguration("File %s already exists!" % path)
fh = None
fh = open(path,'wb')
if LOCK:
fcntl.flock( fh.fileno(), fcntl.LOCK_EX )
aggregationType = struct.pack( longFormat, aggregationMethodToType.get(aggregationMethod, 1) )
oldest = max([secondsPerPoint * points for secondsPerPoint,points in archiveList])
maxRetention = struct.pack( longFormat, oldest )
xFilesFactor = struct.pack( floatFormat, float(xFilesFactor) )
archiveCount = struct.pack(longFormat, len(archiveList))
packedMetadata = aggregationType + maxRetention + xFilesFactor + archiveCount
headerSize = metadataSize + (archiveInfoSize * len(archiveList))
archiveOffsetPointer = headerSize
for secondsPerPoint,points in archiveList:
archiveInfo = struct.pack(archiveInfoFormat, archiveOffsetPointer, secondsPerPoint, points)
archiveOffsetPointer += (points * pointSize)
#If configured to use fallocate and capable of fallocate use that, else
#attempt sparse if configure or zero pre-allocate if sparse isn't configured.
if CAN_FALLOCATE and useFallocate:
remaining = archiveOffsetPointer - headerSize
fallocate(fh, headerSize, remaining)
elif sparse: - 1)
remaining = archiveOffsetPointer - headerSize
chunksize = 16384
zeroes = '\x00' * chunksize
while remaining > chunksize:
remaining -= chunksize
if fh:

The function is doing everything: checking if the file doesn't exist already, opening it, building the structured data, writing this, building more structure, then writing that, etc.

That means that the caller has to give a file path, even if it just wants a whipser data structure to store itself elsewhere. StringIO() could be used to fake a file handler, but it will fail if the call to fcntl.flock() is not disabled – and it is inefficient anyway.

There's a lot of other functions in the code, such as for example setAggregationMethod(), that mixes the handling of the files – even doing things like os.fsync() – while manipulating structured data. This is definitely not a good design, especially for a library, as it turns out reusing the function in different context is near impossible.

Race conditions

There are race conditions, for example in create() (see added comment):

if os.path.exists(path):
raise InvalidConfiguration("File %s already exists!" % path)
fh = None
fh = open(path,'wb')

That code should be:

fh = os.fdopen(, os.O_WRONLY | os.O_CREAT | os.O_EXCL), 'wb')
except OSError as e:
if e.errno = errno.EEXIST:
raise InvalidConfiguration("File %s already exists!" % path)

to avoid any race condition.

Unwanted optimization

We saw earlier the fetch() function that is barely useful, so let's take a look at the file_fetch() function that it's calling.

def file_fetch(fh, fromTime, untilTime, now = None):
header = __readHeader(fh)

The first thing the function does is to read the header from the file handler. Let's take a look at that function:

def __readHeader(fh):
info = __headerCache.get(
if info:
return info
originalOffset = fh.tell()
packedMetadata =
(aggregationType,maxRetention,xff,archiveCount) = struct.unpack(metadataFormat,packedMetadata)
raise CorruptWhisperFile("Unable to read header",

The first thing the function does is to look into a cache. Why is there a cache?

It actually caches the header based with an index based on the file path ( Except that if one for example decide not to use file and cheat using StringIO, then it does not have any name attribute. So this code path will raise an AttributeError.

One has to set a fake name manually on the StringIO instance, and it must be unique so nobody messes with the cache

import StringIO
packedMetadata = <some source>
fh = StringIO.StringIO(packedMetadata) = "myfakename"
header = __readHeader(fh)

The cache may actually be useful when accessing files, but it's definitely useless when not using files. But it's not necessarily true that the complexity (even if small) that the cache adds is worth it. I doubt most of whisper based tools are long run processes, so the cache that is really used when accessing the files is the one handled by the operating system kernel, and this one is going to be much more efficient anyway, and shared between processed. There's also no expiry of that cache, which could end up of tons of memory used and wasted.


None of the docstrings are written in a a parsable syntax like Sphinx. This means you cannot generate any documentation in a nice format that a developer using the library could read easily.

The documentation is also not up to date:

def fetch(path,fromTime,untilTime=None,now=None):
def create(path,archiveList,xFilesFactor=None,aggregationMethod=None,sparse=False,useFallocate=False):

This is something that could be avoided if a proper format was picked to write the docstring. A tool cool be used to be noticed when there's a diversion between the actual function signature and the documented one, like missing an argument.

Duplicated code

Last but not least, there's a lot of code that is duplicated around in the scripts provided by whisper in its bin directory. Theses scripts should be very lightweight and be using the console_scripts facility of setuptools, but they actually contains a lot of (untested) code. Furthermore, some of that code is partially duplicated from the library which is against DRY.


There are a few more things that made me stop considering whisper, but these are part of the whisper features, not necessarily code quality. One can also point out that the code is very condensed and hard to read, and that's a more general problem about how it is organized and abstracted.

A lot of these defects are actually points that made me start writing The Hacker's Guide to Python a year ago. Running into this kind of code makes me think it was a really good idea to write a book on advice to write better Python code!

A book I wrote talking about designing Python applications, state of the art, advice to apply when building your application, various Python tips, etc. Interested? Check it out.

-1:-- Python bad practice, a concrete case (Post Julien Danjou)--L0--C0--September 15, 2014 11:09 AM

Grant Rettke: zk-phi releases two perhaps necessary packages for Emacs

You may not be totally sure why, but you will immediately feel like you ought to install the following two packages for Emacs:

[indent-guide] show vertical lines to guide indentation

[highlight-stages] highlight staged (quasi-quoted) expressions

-1:-- zk-phi releases two perhaps necessary packages for Emacs (Post Grant)--L0--C0--September 12, 2014 01:46 AM

Grant Rettke: Problems installing (M)ELPA packages

In Problems installing (M)ELPA packages Sebastien wrote:

While using `package.el’ fulfills a long-awaited dream for easily adding and upgrading Emacs libraries, I now feel upgrading packages is much more complex than what I anticipated.

At least two of the packages I use daily (Org and Helm) require to be installed from inside a fresh Emacs session (emacs -Q), where nothing has been loaded.

I had wondered but never found out why one must follow this approach when installing org from scratch. Good to know!

-1:-- Problems installing (M)ELPA packages (Post Grant)--L0--C0--September 11, 2014 11:41 PM

Timo Geusch: Install your basic Emacs packages via a single function call

If you, like me tend to carry around or “cloud around” a single .emacs file so you end up with similar environments wherever you have an Emacs install, you know it’s a little painful to ensure that you have the same set of basic packages installed on each one of your Emacs installations. As I […]
-1:-- Install your basic Emacs packages via a single function call (Post Timo Geusch)--L0--C0--September 07, 2014 12:14 AM

Mickey Petersen: Emacs Chat with Sacha Chua

Sacha Chua’s done a great (if I say so myself) video interview with me about Emacs, my thoughts on Emacs, this blog, and how I use Emacs.

Head on over to her blog and have a look!

Be sure to check out the rest of her blog: she’s a real Org mode expert!


-1:-- Emacs Chat with Sacha Chua (Post mickey)--L0--C0--September 05, 2014 08:32 PM

Bozhidar Batsov: clojure-mode 3.0

clojure-mode 3.0 is out!

It’s one of the most ambitious releases in recent times and brings much improved font-locking across the board. Other notable changes include dropping support for Emacs 23 (CIDER doesn’t support it either) and removing some deprecated features (most notably the functionality for switching between code and its test; see Projectile for an awesome replacement of the old feature).

An extensive list of the changes is available here.

This version also marks the introduction of an automated test suite (currently it consists mostly of font-lock tests), which should make it easier to do changes in the future.

Next step – indentation improvements and decoupling clojure-mode from lisp-mode. Both tasks are related. We’ve been deriving much from lisp-mode since day 1 and this has worked reasonably well so far, but the truth is that Clojure is not Common Lisp (or Emacs Lisp for that matter) and would benefit from a more refined syntax table, indentation rules, etc.

When (if) this will happen? Sadly, I have no idea… Help is definitely welcome! If you don’t have the time to help out with code or docs you can still support my work on clojure-mode (and all my other projects) via gittip.

Support via Gittip

That’s all for now, folks! Enjoy the new clojure-mode!

-1:-- clojure-mode 3.0 (Post)--L0--C0--September 05, 2014 12:15 PM

Emacs Redux: Emacs Lisp Style Guide

I’ve started working on a community Emacs Lisp style guide. My goal is to create a comprehensive document similar to the Clojure and Ruby community style guides.

Such a document would be a great starting point for many newcomers to the Emacs community and should increase the overall quality of the code produced by it. There’s much to be documented about code layout, preferred syntactic conventions, major and minor mode definitions, etc. and I obviously can’t know the best practices about everything so I’ll need a lot of help.

Of course, we already have the official coding guidelines, but they are a bit light on examples, rationale, etc. Hopefully the community style guide will manage to fill any gaps in them and in good time portions of it will find their place in the official documentation.

I’m inviting all members of our community interested in promoting good Emacs Lisp practices to join the development of this document!

-1:-- Emacs Lisp Style Guide (Post)--L0--C0--September 04, 2014 12:00 PM

Bozhidar Batsov: SICP Distilled

SICP Distilled is one of the most interesting Kickstarter projects I’ve seen in a while.

Its creator Tom Hall is planning to create some nice companion resources for SICP with code examples in Clojure. In his own words:

It’s a long book, with lots
of exercises and lots of people I know have started, loved it, but
somehow not finished.

Abelson and Sussman themselves highlight the important lessons of SICP
in their paper `Lisp: A Language For Stratified Design` and I have my
own favourite bits.

As the book itself is available online for free I want to make the
perfect accompaniment to it - an ebook summarising the key ideas,
short videos describing them, screencasts of solving some of the
exercises, translation of the examples into Clojure, example projects,
partial solutions for you to complete (similar to `4clojure` and `Clojure
koans`) and a place to discuss solving them with people and hopefully
keep momentum and get it finished!

Something to be enjoyed alongside SICP, rather than completely replace it.

Maybe some ideas come out a little different in Clojure, or I take a
slightly novel approach (hence idiosyncratic), maybe I miss something
out (hence tour, sorry), but I hope we can have some fun along the

Tom Hall SICP Distilled

I’m one of those many people who never did finish SICP (although I hope to do that some day), so I can totally relate to Tom’s words. I’ve already backed his campaign and I hope more of you will do the same!

-1:-- SICP Distilled (Post)--L0--C0--August 29, 2014 03:15 PM

Emacs Redux: A peek at Emacs 24.4: superword-mode

In a previous post I wrote about camel-case aware editing with subword-mode. Emacs 24.4 adds a complementary minor mode called superword-mode, which also alters the behavior of word-based commands when enabled.

Normally Emacs would consider underscores and dashes word separators (snake_case and lisp-case anyone?). This affects all word commands - forward-word, backward-word, kill-word, etc. Let’s see a couple of examples (| denotes the cursor position):

;; word with dash

;; press M-f (forward-word) once

;; press M-f again

;; press M-b (backward-word) once

;; word with underscore

;; press M-f once

;; press M-f again

;; press M-b once

;; word in camelCase (assuming subword-mode is not enabled)

;; press M-f once

;; word in camelCase (assuming subword-mode is enabled)

;; press M-f once

Personally I find the default behavior combined with subword-mode great. I do a lot of Ruby and Lisp programming and it also makes a lot of sense to me to be able to navigate the portions of a complex word, but I guess not everyone feels this way. Enter superword-mode - when it’s enabled all “complex/compound” words are treated as a single word:

;; word with dash

;; press M-f once

;; word with underscore

;; press M-f once

;; word in camelCase

;; press M-f once

Note that you cannot have subword-mode and superword-mode enabled at the same time. Turning one of them on will disable the other.

Personally, I don’t see much value in superword-mode as a mode that’s enabled all the time, but I can imagine some useful scenarios in which I’d enable it briefly to do some focused editing.

-1:-- A peek at Emacs 24.4: superword-mode (Post)--L0--C0--August 27, 2014 03:24 PM

Mickey Petersen: Swapping quote symbols in Emacs with parse-partial-sexp

Ever wonder how Emacs *knows* that the point is next to a balanced expression like a bracket or a string? That kill-sexp or up-list knows how to kill the balanced expression at point or move up one level in a nested set of brackets, in just about any major mode out there?

At a higher level the mode author may choose to write his own parsing routines for handling balanced expressions in that particular mode; but unless your mode’s very special (LaTeX comes to mind) most choose to rely on Emacs’s own ability to parse certain recurring concepts that appear in most major modes — be it something as simple as text-mode or as complicated as a full-blown major mode. A few modes may augment this basic parser with additional functionality, but for a lot of languages (particularly C-like languages) — it’s more than enough.

The secret is parse-partial-sexp and syntax-ppss. The latter function is still an undocumented snippet in syntax.el and yet it’s a first-class citizen used in hundreds of places all over the Emacs elisp source code.

So what makes a barely-kept secret to mode writers such an interesting function? It possesses the unique ability to parse and understand common syntactic elements: comments; strings; and balanced expressions, like brackets. It uses Emacs’s syntax table, a giant key-value table used by modes and some commands to tell Emacs how it should treat alphabetical characters and special characters like [ and ".

So the good news is all that information's already taken care of for you by the mode authors: all you have to do is think of instances where you may want to edit or move by syntax.

So let's say you want to switch the quote symbols in Python from single to double-quote or vice versa:

def foobar():
    s = 'Hello█ World'
    return s

We want to run one command that turns 'Hello█ World' into "Hello█ World" where █ is the point.

Because syntax-ppss understands the basics of our major mode's language syntax this is rather easy to do.

Let's write a function that determines if point is in a string or not (actually, such a function already exists in thingatpt.el called in-string-p):

(defun point-in-string-p (pt)
  "Returns t if PT is in a string"
  (eq 'string (syntax-ppss-context (syntax-ppss pt))))

Here I'm using syntax-ppss-context, a helper function that can return three different states: comment, string, or nil.

If you eval M-: (point-in-string-p (point)) you can test if the point in the active buffer is in a string or not.

Making a function that does the same for comments is trivial: replace string with comment and bob's your uncle.

The next piece of the puzzle is that we need to be able to move out of a string if we're in one or throw an error if we are not. We need to "move out" of it as we want to replace the quotes surrounding a string; the easiest way to do this reliably is to find the bounds of the string: the beginning and end.

(defun beginning-of-string ()
  "Moves to the beginning of a syntactic string"
  (unless (point-in-string-p (point))
    (error "You must be in a string for this command to work"))
  (while (point-in-string-p (point))
    (forward-char -1))

This function is marked as interactive so it can be called through M-x, bound to a key, used in macros, and so forth. Then we test if we're not in a string: if we are not, we bail out with an error. You could remove that check and rely on the while loop failing the initial condition but then you wouldn't get an error message that would propagate.

The while loop itself is simple: as long as we're in a string go forward by negative one characters (that's the elisp convention for going backwards.) And finally we return the location of point.

The next step is to go to the end of the string. We can do that in two ways: extend beginning-of-string so it's generic and will take a direction: -1 for backwards and 1 for forwards. The other is to use Emacs's own set of commands that work on s-expressions, like forward-sexp (C-M-f.)

The latter is easier (and the former left as an exercise to the reader.) What we need now are the points immediately before and after each quote symbol before we start changing things.

If we change the first quote symbol at the beginning of the string then we are left with invalid syntax and Emacs's parse-partial-sexp cannot reconcile the two mismatched quotes when we call forward-sexp. So we have to store the positions first, and then change the quotes.

The other thing we have to remember is to move the point back to the original position where the user called the command; not doing so is considered bad form in elisp land: you should not move the point unless the point (ha) of the command is to move around the buffer.

(defun swap-quotes ()
  "Swaps the quote symbols in a \\[python-mode] string"
    (let ((bos (save-excursion
          (eos (save-excursion
          (replacement-char ?\'))
      (goto-char bos)
      ;; if the following character is a single quote then the
      ;; `replacement-char' should be a double quote.
      (when (eq (following-char) ?\')
          (setq replacement-char ?\"))
      (delete-char 1)
      (insert replacement-char)
      (goto-char eos)
      (delete-char -1)
      (insert replacement-char))))

This interactive command will swap the quotes of the string point is in. It starts out by recording the position of the beginning and end of the string. If we're not in a string, the command will exit with the error propagated from the beginning-of-string command above.

Next we go to the beginning of the string and we check what the replacement character should be and delete the next character and insert our replacement character. The same is repeated for the end of the string. And finally because we save-excursion at the beginning of the command the point is placed back at its original position.

There are a few obvious improvements: the delete/insertion code could be abstracted into a letf-bound function - but that seems like overkill. Another optimization is supporting triple quotes by using the looking-at function and passing it a regular expression that matches single or double quotes, in singles or triples, and replacing the match with the replacement character.

But the function works. And it could easily work for other modes with interchangeable quotes -- or even other paired expressions, like brackets. In fact, making it work with brackets is easier as the built-in command up-list (C-M-u) will go to the beginning of a balanced pair and forward-sexp will go to the end of the balanced pair at point.

Thanks to a little-known feature of Emacs's syntax parser you can make some simple assumptions about the text in a buffer and act on it in a structured manner.


-1:-- Swapping quote symbols in Emacs with parse-partial-sexp (Post mickey)--L0--C0--August 26, 2014 03:18 PM

Chris Ball: Experimenting with Panamax

Disclosure: This blog post is part of the Panamax Template Contest.

In my blog post about the Dell C6100 server I’ve been using, I mentioned that I run a full LXC userland for each application I deploy, and that I’d like to try out Docker but that this setup is in conflict with Docker’s philosophy – a Docker container only runs one process, which makes it difficult to use Docker for anything requiring interaction between processes. Here’s an example: this blog is running WordPress with MySQL. So, with LXC I create a fresh Ubuntu container for the blog and run apt-get install wordpress and I’m up and running, but trying to use Docker would leave me with an “orchestration” problem – if I’m supposed to have a separate web server and database server, how will they figure out how to talk to each other?

If the two Docker services are being run on the same host, you can use docker --link, which runs one service under a given name and then makes it available to any service it’s linked to. For example, I could call a postgres container db and then run something like docker --name web --link db:db wordpress. The wordpress container receives environment variables giving connection information for the database host, which means that as long as you can modify your application to use environment variables when deciding which database host to connect to, you’re all set. (If the two docker services are being run on separate hosts, you have an “ambassador” problem to figure out.)

All of which is a long-winded way to say that Panamax is a new piece of open source software that attempts to ameliorate the pain of solving orchestration problems like this one, and I decided to try it out. It’s a web service that you run locally, and it promises a drag-and-drop interface for building out complex multi-tier Docker apps. Here’s what it looks like when pairing a postgres database with a web server running a Django app, WagtailCMS:

The technical setup of Panamax is interesting. It’s distributed as a CoreOS image which you run inside Vagrant and Virtualbox, and then your containers are launched from the CoreOS image. This means that Panamax has no system dependencies other than Vagrant and Virtualbox, so it’s easily usable on Windows, OS X, or any other environment that can’t run Docker directly.

Looking through the templates already created, I noticed an example of combining Rails and Postgres. I like Django, so I decided to give Django and Postgres a try. I found mbentley’s Ubuntu + nginx + uwsgi + Django docker image on the Docker Hub. Comparing it to the Rails and Postgres template on Panamax, the Django container lacks database support, but does have support for overlaying your own app into the container, which means you can do live-editing of your app.

I decided to see if I could combine the best parts of both templates to come up with a Panamax template for hosting arbitrary Django apps, which supports using an external database and offers live-editing.  I ended up creating a new Docker image, with the unwieldy name of cjbprime/ubuntu-django-uwsgi-nginx-live. This image is based on mbentley’s, but supports having a Django app passed in as an image, and will try to install its requirements. You can also link this image to a database server, and syncdb/migrate will be run when the image starts to set things up. If you need to create an admin user, you can do that inside a file in your app directory.

After combining this new Docker image with a Postgres container, I’m very happy with how my django-with-postgres template turned out – I’m able to take an existing Django app, make minor changes using a text editor on my local machine to use environment variables for the database connection, start up the Panamax template, and watch as a database is created (if necessary), dependencies are installed, migrations are run, an admin user is created (if necessary), and the app is launched.  All without using a terminal window at any point in the process.

To show a concrete example, I also made a template that bundles the Wagtail Django CMS demo. It’s equivalent to just using my django-with-postgres container with the wagtaildemo code passed through to the live-editing overlay image (in /opt/django/app), and it brings up wagtaildemo with a postgres DB in a separate container. Here’s what that looks like:

Now that I’ve explained where I ended up, I should talk about how Panamax helped.  Panamax introduced me to Docker concepts (linking between containers, overlaying images) that I hadn’t used before because they seemed too painful, and helped me create something cool that I wouldn’t otherwise have attempted.  There were some frustrations, though.  First, the small stuff:

Underscores in container names

This one should have been in big bold letters at the top of the release notes, I think.  Check this out: unit names with _{a-f}{a-f} in them cause dbus to crash. This is amusing in retrospect, but was pretty inscrutable to debug, and perhaps made worse by the Panamax design: there’s a separate frontend web service and backend API, and when the backend API throws an error, it seems that the web interface doesn’t have access to any more detail on what went wrong. I’m lucky that someone on IRC volunteered the solution straight away.

The CoreOS Journal box occasionally stays black

Doing Docker development depends heavily on being able to see the logs of the running containers to work out why they aren’t coming up as you thought they would.  In Docker-land this is achieved with docker -f logs <cid>, but Panamax brings the logs in to the web interface: remember, the goal is to avoid having to look at the terminal at all.  But it doesn’t work sometimes.  There’s a panamax ssh command to ssh into the CoreOS host and run docker logs there, but that’s breaking the “fourth wall” of Panamax.

Progress bar when pulling Docker images

A minor change: it’d be great to be able to see progress when Panamax is pulling down a Docker image. There’s no indicator of progress, which made me think that something had hung or failed. Further, systemd complained about the app failing to start, when it just needed more time for the docker pull to complete.

Out of memory when starting a container

The CoreOS host allocates 1GB RAM for itself: that’s for the Panamax webapp (written in Rails), its API backend, and any containers you write and launch.  I had to increase this to 2GB while developing, by modifying ~/.panamax/.env:

export PMX_VM_MEMORY=2048

Sharing images between the local host and the container

I mentioned how Panamax uses a CoreOS host to run everything from, and how this drastically reduces the install dependencies.  There’s a significant downside to this design – I want to allow my local machine to share a filesystem and networking with my Docker container, but now there’s a CoreOS virtual machine in the way – I can’t directly connect from my laptop to the container running Django without hopping through the VM somehow. I want to connect to it for two different reasons:

  1. To have a direct TCP connection from my laptop to the database server, so that I can make database changes if necessary.
  2. To share a filesystem with a container so that I can test my changes live.

Panamax makes the first type of connection reasonably easy. There’s a VirtualBox command for doing port forwarding from the host through to the guest – the guest in this case is the CoreOS host. So we end up doing two stages of port forwarding: Docker forwards port 80 from the Django app out to port 8123 on the CoreOS host, and then VirtualBox forwards port 8123 on my laptop to port 8123 on the CoreOS host. Here’s the command to make it work:

VBoxManage controlvm panamax-vm natpf1 rule1,tcp,,8123,,8123

The filesystem sharing is much trickier – we need to share a consistent view of a single directory between three hosts: again, the laptop, the CoreOS host, and the Docker app. Vagrant has a solution to this, which is that it can NFS share a guest OS from the CoreOS host back to my laptop. That works like this, modifying ~/.vagrant.d/boxes/panamax-coreos-box-367/0/virtualbox/Vagrantfile: "private_network", ip: ""
  config.vm.synced_folder "/home/cjb/djangoapp", "/home/core/django",
  id: "django", :nfs => true, :mount_options => ['nolock,vers=3,udp']

So, we tell Panamax to share /opt/django/app with the CoreOS host as /home/core/django, and then we tell Vagrant to share /home/cjb/djangoappon my laptop with the CoreOS host as /home/core/django over NFS. After `apt-get install nfs-kernel-server`, trying this leads to a weird error:

exportfs: /home/cjb/djangoapp does not support NFS export

This turns out to be because I’m running ecryptfs for filesystem encryption on my Ubuntu laptop, and nfs-kernel-server can’t export the encrypted FS. To work around it, I mounted a tmpfs for my Django app and used that instead. As far as I know, OS X and Windows don’t have this problem.


Panamax taught me a lot about Docker, and caused me to publish my first two images to the Docker registry, which is more than I expected to gain from trying it out. I’m not sure I’m the target audience – I don’t think I’d want to run production Docker apps under it on a headless server (at least until it’s more stable), which suggests that its main use is as an easy way to experiment with the development of containerized systems. But the friction introduced by the extra CoreOS host seems too great for it to be an awesome development platform for me. I think it’s a solvable problem – if the team can find a way to make the network port forwarding and the filesystem NFS sharing be automatic, rather than manual, and to work with ecryptfs on Ubuntu, it would make a massive difference.

I am impressed with the newfound ability to help someone launch a database-backed Django app without using any terminal commands, even if they’re on Windows and have no kind of dev environment, and would consider recommending Panamax for someone in that situation. Ultimately, maybe what I’ll get out of Panamax is a demystification of Docker’s orchestration concepts. That’s still a pretty useful experience to have.

-1:-- Experimenting with Panamax (Post cjb)--L0--C0--August 25, 2014 02:35 PM

Tom Tromey: Emacs verus notification area, again

Ages and ages I wrote about letting Emacs code access the notification area.  I have more to say about it now, but first I want to bore you with some rambling thoughts and some history.

The “notification area” is also called the “status icon area” or the “systray” — it is a spot that holds some icons that are under control of various applications.

I was a fan of the notification area since it first showed up in Gnome.  I recognized it instantly as the thing I wanted that I hadn’t realized I wanted.

Now, as you know, the notification area has fallen on hard times.  It’s been removed in Gnome 3… I searched a bit for the rationale for this deletion, which as far as I can tell is just that some applications abused it, whatever that means; or that it was used inconsistently, which I think the web has conclusively proven is fine by users.  Coming from the Emacs perspective, where one can customize the somewhat-equivalent of the status area (see those recent posts on diminishing minor-mode lighters in the mode line…), and where a certain amount of per-mode idiosyncrasy is the norm, these seem like an inadequate reasons.

However, the reason doesn’t really matter.  I love the notification area!  When I moved more of my daily desktop use back into Emacs (the tides are strong but slow, and take years to come in or go out), I hooked Emacs up to it, and made it a part of my basic configuration.

It’s indispensable now.  What I particularly like about it is that it is both noticeable and unobtrusive — the former because I can have the icons blink on important events, and the latter because the icons don’t move around or obscure other windows.

Ok!  You should use it!  And I totally plan to tell you how, but first some boring history.

My original post relied on a hacked version of the Gnome zenity utility.  This turned out to be a real pain over time.  I had to rebuild it periodically, adding hacks (once removing chunks), etc.  Sharing it with others was hard.  And, for whatever reason, the patches in Gnome bugzilla were completely ignored.  Bah.

A bit later I wrote a big patch to Emacs to put all this into the core.  That patch was rejected, more or less.  Bah two.

Then even later I flirted with KDE for a bit.  Yes.  KDE had the nice idea to expose the notification area via dbus, and Emacs could talk dbus… so I did the obvious thing in elisp.  However the KDE notification area was pretty buggy and in the end I had to abandon it as well.

So, it was back to zenity… until this week, during my funemployment.  I rewrote my hacks in Python.  This was so easy I wish I’d done it years and years ago.

I’m not sure what the moral of this story is.  Maybe that my obsession is your gain.  Or maybe that I have trouble letting go.

Anyway, the result is here, on github, or in marmalade.  You’ll need Python and the new (introspection-based) Python Gtk interfaces.  This of course is no trouble to install.  The package includes the base status icon API, plus basic UIs for ERC and EMMS.  Try it out and let me know what you think.

-1:-- Emacs verus notification area, again (Post tom)--L0--C0--August 25, 2014 03:29 AM

Timo Geusch: Unique buffer names in Emacs

A common annoyance with Emacs when working on a code base that has duplicate file names is that the mode line tends to display the buffer names as “<1>”, “<2>” etc etc. That doesn’t help much with telling them apart and I find it confusing. I was introduced to the Uniquify library a while ago. […]
-1:-- Unique buffer names in Emacs (Post Timo Geusch)--L0--C0--August 23, 2014 10:58 PM

Tom Tromey: Another Mode Line Hack

While streamlining my mode line, I wrote another little mode-line feature that I thought of ages ago — using the background of the mode-line to indicate the current position in the buffer. I didn’t like this enough to use it, but I thought I’d post it since it was a fun hack.

First, make sure the current mode line is kept:

(defvar tromey-real-mode-line-format mode-line-format)

Now, make a little function that format the mode line using the standard rules and then applies a property depending on the current position in the buffer:

(defun tromey-compute-mode-line ()
  (let* ((width (frame-width))
     (line (substring 
        (concat (format-mode-line tromey-real-mode-line-format)
            (make-string width ? ))
        0 width)))
    ;; Quote "%"s.
    (setq line
      (mapconcat (lambda (c)
               (if (eq c ?%)
             ;; It's absurd that we must wrap this.
             (make-string 1 c)))
             line ""))

    (let ((start (window-start))
      (end (or (window-end) (point))))
      (add-face-text-property (round (* (/ (float start)
                    (length line)))
                  (round (* (/ (float end)
                    (length line)))
                  'region nil line))

We have to do this funny wrapping and “%”-quoting business here because the :eval form returns a mode line format — not just text — and because the otherwise appealing :propertize form doesn’t allow computations.

Also, I’ve never understood why mapconcat can’t handle a character result from the map function.  Anybody?

Now set this to be the mode line:

(setq-default mode-line-format '((:eval (tromey-compute-mode-line))))

The function above changes the background of the mode line corresponding to the current window’s start and end positions.  So, for example, here we are in the middle of a buffer that is bigger than the window:

Screenshot - 08222014 - 12:52:19 PM

I left this on for a bit but found it too distracting.  If you like it, use it. You might like to remove the mode-line-position stuff from the mode line, as it seems redundant with the visual display.

-1:-- Another Mode Line Hack (Post tom)--L0--C0--August 22, 2014 07:05 PM

Eric James Michael Ritz: Emacs: No More of My Own Code for PHP Mode

My apologies for the quiet month, as I’ve plans weighing on my mind. In the past I have asked for another Emacs Lisp developer to hopefully step up and offer to maintain PHP Mode. As of today I will no longer work on feature requests for the project. If developers continue to send me improvements and/or bug fixes then I will be happy to review and merge them as necessary. However, I feel like it is time to end my personal work on PHP Mode. I will not completely walk away and leave the project in a dead state until someone is willing to take over the role as maintainer. But from today forward I will no longer be an active contributor, instead simply merging contributions from the myriad of developers who have worked to improve PHP Mode to what it is today.

If you are a PHP programmer, use Emacs, and wish to improve the mode then please take a loot at the latest mode. If you have any questions feel free to ask any quenions.

-1:-- Emacs: No More of My Own Code for PHP Mode (Post ericjmritz)--L0--C0--August 22, 2014 04:54 PM

Jorgen Schäfer: Circe 1.4 released

I just released version 1.4 of Circe, a Client for IRC in Emacs.

The package is available from github and MELPA unstable, even though the latter will track further development changes, so use at your own risk.

Due to the sorry state of Emacs Lisp package archives, I am currently unable to do an actual release of Circe.


  • Non-blocking TLS connections. Circe now supports secure TLS connections to IRC servers without blocking during the connect. (Thanks to defanor for the patches!)
  • PEP (Python Enhancement Proposals) mentions are now buttonized by default, like RFCs and SRFIs.
  • URLs are now buttonized with the logic from the built-in thingatpt.el library.
  • Topic changes are highlighted correctly. Before, Circe would show diffs for topics one older than the last.
  • Circe will now try a random nick when the server says that the initial nick is erraneous. This could happen when the server set a collision-avoiding nick, and Circe tried to retain it after a reconnect.
  • The lui-logging module can now easily be enabled or disabled globally.
  • The new option circe-server-buffer-name can be used to influence the name of the server buffer, especially useful for people using the same bouncer for multiple networks. This can also be set on a per-network basis.
-1:-- Circe 1.4 released (Post Jorgen Schäfer ( 22, 2014 04:24 PM

Alex Schroeder: Emacs and Dice

Somebody asked on Google+: What are the odds for rolling 2d10 and a d20 and getting higher on the 2d10?

(let ((a 0) (b 0) (n 0))
  (insert "2d10 vs. 1d20")
  (insert "     ")
  (dotimes (roll-1 10)
    (dotimes (roll-2 10)
      (insert (format " %5s"
		      (concat (number-to-string (1+ roll-1)) "+"
			      (number-to-string (1+ roll-2)))))))
  (dotimes (roll-3 20)
    (insert (format " %4d" (1+ roll-3)))
    (dotimes (roll-1 10)
      (dotimes (roll-2 10)
	(insert "     "
		(cond ((> (+ 2 roll-1 roll-2) (1+ roll-3))
		       (incf a)
		      ((= (+ 2 roll-1 roll-2) (1+ roll-3))
		       (incf n)
		       (incf b)
  (let ((total (+ a b n)))
    (insert (format "A: %d / %d = %4.3f\n" a total (/ a 1.0 total)))
    (insert (format "B: %d / %d = %4.3f\n" b total (/ b 1.0 total)))
    (insert (format "-: %d / %d = %4.3f\n" n total (/ n 1.0 total))))

2d10 vs. 1d20

        1+1   1+2   1+3   1+4   1+5   1+6   1+7   1+8   1+9  1+10   2+1   2+2   2+3   2+4   2+5   2+6   2+7   2+8   2+9  2+10   3+1   3+2   3+3   3+4   3+5   3+6   3+7   3+8   3+9  3+10   4+1   4+2   4+3   4+4   4+5   4+6   4+7   4+8   4+9  4+10   5+1   5+2   5+3   5+4   5+5   5+6   5+7   5+8   5+9  5+10   6+1   6+2   6+3   6+4   6+5   6+6   6+7   6+8   6+9  6+10   7+1   7+2   7+3   7+4   7+5   7+6   7+7   7+8   7+9  7+10   8+1   8+2   8+3   8+4   8+5   8+6   8+7   8+8   8+9  8+10   9+1   9+2   9+3   9+4   9+5   9+6   9+7   9+8   9+9  9+10  10+1  10+2  10+3  10+4  10+5  10+6  10+7  10+8  10+9 10+10
    1     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    2     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    3     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    4     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    5     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    6     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    7     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    8     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    9     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
   10     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
   11     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A
   12     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A
   13     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A
   14     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A
   15     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A
   16     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A
   17     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A
   18     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A
   19     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A
   20     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -

A: 1000 / 2000 = 0.500
B: 900 / 2000 = 0.450
-: 100 / 2000 = 0.050

So, the odds of rolling higher on 2d10 is 50%.

(Does anybody have some actual math on how to do this?)


-1:-- Emacs and Dice (Post)--L0--C0--August 22, 2014 07:25 AM

M-x all-things-emacs: The Editor of a Lifetime

Perry Metzger has been using Emacs as his text editor since early September, 1983 — nearly 31 years. Over much of that time, it has also been his primary way to read email, compile programs, and perform a variety of other tasks.
Why would anyone use a single program for that long? This talk is partially intended to answer that question.
Emacs remains one of the most important user interfaces (and text editors) for computer professionals almost 40 years after it was created. The talk is intended to be part history, part philosophy, and part speculation on the future. It will also teach Emacs fans how to explain to their skeptical friends why it is still a good idea to learn a tool from the terminal era that requires memorization of dozens of control sequences in an age of GUIs and smart phones.
-1:-- The Editor of a Lifetime (Post Ryan McGeary)--L0--C0--August 21, 2014 04:47 PM

Got Emacs?: Emacs fourth pretest released

Missed the announcement but the fourth pretest is now available for testing.
-1:-- Emacs fourth pretest released (Post sivaram ( 20, 2014 05:55 PM

Julien Danjou: Tracking OpenStack contributions in GitHub

I've switched my Git repositories to GitHub recently, and started to watch my contributions statistics, which were very low considering I spend my days hacking on open source software, especially OpenStack.

OpenStack hosts its Git repositories on its own infrastructure at, but also mirrors them on GitHub. Logically, I was expecting GitHub to track my commits there too, as I'm using the same email address everywhere.

It turns out that it was not the case, and the help page about that on GitHub describes the rule in place to compute statistics. Indeed, according to GitHub, I had no relations to the OpenStack repositories, as I never forked them nor opened a pull request on them (OpenStack uses Gerrit).

Starring a repository is enough to build a relationship between a user and a repository, so this is was the only thing needed to inform GitHub that I have contributed to those repositories. Considering OpenStack has hundreds of repositories, I decided to star them all by using a small Python script using pygithub.

And voilà, my statistics are now including all my contributions to OpenStack!

-1:-- Tracking OpenStack contributions in GitHub (Post Julien Danjou)--L0--C0--August 19, 2014 05:00 PM

Alex Schroeder: Gnus Mystery Emails

These days I have a Raspberry Pi running Dovecot running as my IMAP Server. I connect to this IMAP Server from my iPhone using Apple’s Mail and from my Laptop using Gnus. For a while now I have noticed that the two don’t list the same mails in my INBOX. In Gnus, I enter the group using C-u RET or I use / o to insert old articles and I see 14 mails. When I enter the Server buffer from the Group buffer using ^ and switch to the INBOX, I see 72 mails. What could be the reason for this?


-1:-- Gnus Mystery Emails (Post)--L0--C0--August 14, 2014 07:54 AM

Bryan Murdock: Free Verilog Simulators

At DVCon 2013 I asked JL Gray's panel if we would ever have Free tools, like the software world. None of panelists seemed to think so, one of the panelists, a Mentor employee, scoffed, "you get what you pay for with free tools." Never mind that their (and Cadence's and Synopsys's) products are very likely developed with tools that contain millions of lines of Free software.

So, to work towards answering my own question, I spent a little time and looked for Free/Open Source verilog simulators. Here's what I found:

Icarus Verilog

GPL Cver

PVSim Verilog Simulator

VeriWell Verilog Simulator

I have personally used Icarus and Cver before, but not very extensively.  They were usable and seemed pretty complete, for Verilog.  None of the above claim any support of SystemVerilog except for Icarus.  The Icarus developer at one point expressed abhorrence at SystemVerilog but it seems support for some parts of the language have been added.

PVSim and VeriWell were new to me.  I'll give them a try, hopefully soon, and post more information.

Another one that should be mentioned is Verilator.  I have downloaded and played with this one too.  It only supports synthesizable Verilog, so have fun writting a testbench.  I think the intent is for you to write your testbench in C++, so if you like that idea then this could be a good one to try too.

Did I miss any?

UPDATE: All of these (except Verilator) plus a host of other free EDA tools are available to easily try out at EDA Playground. Go there now, it's a fun place to play.

2ND UPDATE: As mentioned in the comments below, there are two other ways to try Icarus online: and

-1:-- Free Verilog Simulators (Post Bryan ( 12, 2014 02:31 PM

Jorgen Schäfer: Elpy 1.5.1 released

I just released version 1.5.1 of Elpy, the Emacs Python Development Environment. This is a bug fix release.

Elpy is an Emacs package to bring powerful Python editing to Emacs. It combines a number of other packages, both written in Emacs Lisp as well as Python.

Quick Installation

Evaluate this:

(require 'package)
(add-to-list 'package-archives
'("elpy" .

Then run M-x package-install RET elpy RET.

Finally, run the following (and add them to your .emacs):


Changes in 1.5.1

  • Fix a bug where company-mode might get confused about the current backend, leading to an error about Symbol’s function definition is void: nil
  • Fix Rope so it won’t search the whole project directory. This was an intended feature in v1.5 which did not work originally.
  • Use yas-text instead of text in snippets for compatibility with the unreleased yasnippet from MELPA (thanks to Daniel Wu!)
-1:-- Elpy 1.5.1 released (Post Jorgen Schäfer ( 10, 2014 07:23 AM

John Sullivan: No such pipe, or this pipe has been deleted

This data comes from but the Pipe does not exist or has been deleted.
-1:-- No such pipe, or this pipe has been deleted (Post)--L0--C0--August 03, 2014 02:58 PM

Flickr tag 'emacs': 2014-08-03_032527_1366x768_scrot

Thiago Perrotta posted a photo:


Example of how to use org mode to manage a project.

-1:-- 2014-08-03_032527_1366x768_scrot (Post Thiago Perrotta ( 03, 2014 06:27 AM

Flickr tag 'emacs': enviando-y-recibiendo

El Diego Efe posted a photo:


-1:-- enviando-y-recibiendo (Post El Diego Efe ( 31, 2014 08:26 PM

Eric James Michael Ritz: GNU Emacs: Competing With Other Editors—Should It?

Today while browsing the Emacs Reddit I came across this interesting thread. The sensationlist title contains the following quote from co-maintainer Stefan Monnier:

We’re not in the business of competing.

He is referring to competing with other editors and IDEs. Today I want to frame his comment in the proper context and discuss whether GNU Emacs should compete with the likes of Visual Studio and Eclipse.

The Context

Stefen’s comments come from a thread discussing the policy for packages in ELPA, the official Emacs package repository. In the emacs-24 branch of the official repository, the Python mode has going through various non-bug–fix changes. Some people believe this should not happen. Changes to language-specific modes should adhere to the same feature freeze as everything else. Other people believe that changes to such modes should always be welcome, assuming that don’t break anything. One argument for this is that it helps Emacs stay up to do with editors that pump out new features, and this would help Emacs keep up.

And here came the, “We’re not in the business of competing comment.”

The Benefits of Not Competing

GNU Emacs has long done its own thing. Any long-time user will tell you that. This attitude allows the Emacs developers to focus on what interests them without being beholden to the trends and practices of editors such as Visual Studio. Granted—a lot of Emacs Lisp developers will simply ‘steal’ those features be re-writing them in Emacs Lisp, but that’s beside my point.

Decades of this attitude has created an atmosphere around Emacs where newcomers are expected to conform to the standards of Emacs, or to move on to something else. “Why isn’t Control+S ‘save’ instead of starting a search?” I’ve seen these questions a lot. Emacs fans tend to trot out three responses:

  1. You can bind the keys to whatever you want. (Non-obvious to most on how to do that.)

  2. Emacs was that way long before Control+S because the standard for save. (Clinging to the past in an effort to avoid change.)

  3. You’ll get used to it soon enough. (The weakest of all rebuttals.)

It’s worth mentioning CUA Mode helps these issues. But then you have the long-term users who seem to view CUA Mode with the same disdain that the Brazilian national football teams has when it comes to guarding their goal against Germans.

Long Term Objectives

I saw the following quote in the discussion linked above. It in no way reflects the Free Software Foundation. That said, this is not the first time I’ve seen this?

They do not want to succeed in any of these criteria. Users, developers, and donations are nice to have, but simply not a priority of GNU and the FSF.

The only thing they really want to succeed at is freedom, and by their measures Emacs already succeeded in this aspect, being a free software project under a strong copyleft license, owned by a non-profit foundation.

And that freedom does not fade or decrease if Emacs fail to attract users, developers or donations. The license ensures that Emacs is going to remain free as long as our legal system persists, even if no one was left using or developing it.

The Hell…? As another poster succintly put it, “Freedom without people, that’s an interesting goal.”

The Free Software Foundation (FSF) cannot have an attitude like this. They must realize that they compete with other editors whether they want to or not. Many programmers seem to make their choice in editors after trying each for about fifteen to thirty minutes in my experience. Emacs should do whatever it can to rope in new users within that time frame. It’s esoteric behavior that made sense decades ago no longer stands against todays standards.

Now then, the power to customize Emacs is second-to-none. But there is no reason to expect a new user approach Emacs to even realize that. Even worse, some users feel like such new users are hurting the community. They would rather have Emacs as an inclusive club for people who have invested the effort in its arcane arts.


How to change this community culture? I will write my thoughts in the future. But I admit to having no simple answers. If you have any thoughts, ideas, or disagreements then please leave them in the comments.

-1:-- GNU Emacs: Competing With Other Editors—Should It? (Post ericjmritz)--L0--C0--July 26, 2014 11:28 PM