Irreal: Extending Isearch

Nicolas Petton presents a nice bit of Elisp that extends the isearch functionality. With his code, you can search for the symbol at point or the active region if there is one.

The first of these is easily accomplished using【Ctrl+w】from within isearch but that takes an extra key chord. With Petton’s code you just invoke his isearch-thing function and it searches for the symbol at point unless there is an active region.

If you are a heavy user of isearch and find yourself typing 【Ctrl+s】 and then 【Ctrl+w】 a lot, this may be win for you.

-1:-- Extending Isearch (Post jcs)--L0--C0--August 29, 2014 11:44 AM

Irreal: Tramp with Multiple Hops

The Emacs Tramp package is a wonderful thing. With it, you can easily edit files on remote machines in a virtually transparent fashion. Most Emacs users are familiar with Tramp and use it to edit files on machines on the local network or on remote-network machines directly reachable on the Internet. Often times, though, those remote-network machines are not directly reachable.

A typical example occurs when a developer is working from home and needs to access a machine on his work network that is protected by a firewall. Emacs, of course, can also handle this case. The normal way is to set rules for how to reach remote-network hosts in your .emacs or init.el file. The rules are a bit complex but are documented in the Tramp manual. Sometimes, though, you just want a one-off access to some protected host and it’s a pain to temporarily add Tramp configuration rules for it.

Fortunately, David Vázquez has tweeted the answer:

Unless you need to repeatedly access some remote host, this is a much easier method. The tweet tells you just about everything you need to know but you can see the details in the Tramp documentation.

-1:-- Tramp with Multiple Hops (Post jcs)--L0--C0--August 28, 2014 12:06 PM

Emacs Redux: A peek at Emacs 24.4: superword-mode

In a previous post I wrote about camel-case aware editing with subword-mode. Emacs 24.4 adds a complementary minor mode called superword-mode, which also alters the behavior of word-based commands when enabled.

Normally Emacs would consider underscores and dashes word separators (snake_case and lisp-case anyone?). This affects all word commands - forward-word, backward-word, kill-word, etc. Let’s see a couple of examples (| denotes the cursor position):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
;; word with dash
|some-word

;; press M-f (forward-word) once
some|-word

;; press M-f again
some-word|

;; press M-b (backward-word) once
some-|word

;; word with underscore
|some_word

;; press M-f once
some|_word

;; press M-f again
some_word|

;; press M-b once
some_|word

;; word in camelCase (assuming subword-mode is not enabled)
|someWord

;; press M-f once
someWord|

;; word in camelCase (assuming subword-mode is enabled)
|someWord

;; press M-f once
some|Word

Personally I find the default behavior combined with subword-mode great. I do a lot of Ruby and Lisp programming and it also makes a lot of sense to me to be able to navigate the portions of a complex word, but I guess not everyone feels this way. Enter superword-mode - when it’s enabled all “complex/compound” words are treated as a single word:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
;; word with dash
|some-word

;; press M-f once
some-word|

;; word with underscore
|some_word

;; press M-f once
some_word|

;; word in camelCase
|someWord

;; press M-f once
someWord|

Note that you cannot have subword-mode and superword-mode enabled at the same time. Turning one of them on will disable the other.

Personally, I don’t see much value in superword-mode as a mode that’s enabled all the time, but I can imagine some useful scenarios in which I’d enable it briefly to do some focused editing.

-1:-- A peek at Emacs 24.4: superword-mode (Post)--L0--C0--August 27, 2014 03:24 PM

Mickey Petersen: Swapping quote symbols in Emacs with parse-partial-sexp

Ever wonder how Emacs *knows* that the point is next to a balanced expression like a bracket or a string? That kill-sexp or up-list knows how to kill the balanced expression at point or move up one level in a nested set of brackets, in just about any major mode out there?

At a higher level the mode author may choose to write his own parsing routines for handling balanced expressions in that particular mode; but unless your mode’s very special (LaTeX comes to mind) most choose to rely on Emacs’s own ability to parse certain recurring concepts that appear in most major modes — be it something as simple as text-mode or as complicated as a full-blown major mode. A few modes may augment this basic parser with additional functionality, but for a lot of languages (particularly C-like languages) — it’s more than enough.

The secret is parse-partial-sexp and syntax-ppss. The latter function is still an undocumented snippet in syntax.el and yet it’s a first-class citizen used in hundreds of places all over the Emacs elisp source code.

So what makes a barely-kept secret to mode writers such an interesting function? It possesses the unique ability to parse and understand common syntactic elements: comments; strings; and balanced expressions, like brackets. It uses Emacs’s syntax table, a giant key-value table used by modes and some commands to tell Emacs how it should treat alphabetical characters and special characters like [ and ".

So the good news is all that information's already taken care of for you by the mode authors: all you have to do is think of instances where you may want to edit or move by syntax.

So let's say you want to switch the quote symbols in Python from single to double-quote or vice versa:

def foobar():
    s = 'Hello█ World'
    return s

We want to run one command that turns 'Hello█ World' into "Hello█ World" where █ is the point.

Because syntax-ppss understands the basics of our major mode's language syntax this is rather easy to do.

Let's write a function that determines if point is in a string or not (actually, such a function already exists in thingatpt.el called in-string-p):

(defun point-in-string-p (pt)
  "Returns t if PT is in a string"
  (eq 'string (syntax-ppss-context (syntax-ppss pt))))

Here I'm using syntax-ppss-context, a helper function that can return three different states: comment, string, or nil.

If you eval M-: (point-in-string-p (point)) you can test if the point in the active buffer is in a string or not.

Making a function that does the same for comments is trivial: replace string with comment and bob's your uncle.

The next piece of the puzzle is that we need to be able to move out of a string if we're in one or throw an error if we are not. We need to "move out" of it as we want to replace the quotes surrounding a string; the easiest way to do this reliably is to find the bounds of the string: the beginning and end.

(defun beginning-of-string ()
  "Moves to the beginning of a syntactic string"
  (interactive)
  (unless (point-in-string-p (point))
    (error "You must be in a string for this command to work"))
  (while (point-in-string-p (point))
    (forward-char -1))
  (point))

This function is marked as interactive so it can be called through M-x, bound to a key, used in macros, and so forth. Then we test if we're not in a string: if we are not, we bail out with an error. You could remove that check and rely on the while loop failing the initial condition but then you wouldn't get an error message that would propagate.

The while loop itself is simple: as long as we're in a string go forward by negative one characters (that's the elisp convention for going backwards.) And finally we return the location of point.

The next step is to go to the end of the string. We can do that in two ways: extend beginning-of-string so it's generic and will take a direction: -1 for backwards and 1 for forwards. The other is to use Emacs's own set of commands that work on s-expressions, like forward-sexp (C-M-f.)

The latter is easier (and the former left as an exercise to the reader.) What we need now are the points immediately before and after each quote symbol before we start changing things.

If we change the first quote symbol at the beginning of the string then we are left with invalid syntax and Emacs's parse-partial-sexp cannot reconcile the two mismatched quotes when we call forward-sexp. So we have to store the positions first, and then change the quotes.

The other thing we have to remember is to move the point back to the original position where the user called the command; not doing so is considered bad form in elisp land: you should not move the point unless the point (ha) of the command is to move around the buffer.

(defun swap-quotes ()
  "Swaps the quote symbols in a \\[python-mode] string"
  (interactive)
  (save-excursion
    (let ((bos (save-excursion
                 (beginning-of-string)))
          (eos (save-excursion
                 (beginning-of-string)
                 (forward-sexp)
                 (point)))
          (replacement-char ?\'))
      (goto-char bos)
      ;; if the following character is a single quote then the
      ;; `replacement-char' should be a double quote.
      (when (eq (following-char) ?\')
          (setq replacement-char ?\"))
      (delete-char 1)
      (insert replacement-char)
      (goto-char eos)
      (delete-char -1)
      (insert replacement-char))))

This interactive command will swap the quotes of the string point is in. It starts out by recording the position of the beginning and end of the string. If we're not in a string, the command will exit with the error propagated from the beginning-of-string command above.

Next we go to the beginning of the string and we check what the replacement character should be and delete the next character and insert our replacement character. The same is repeated for the end of the string. And finally because we save-excursion at the beginning of the command the point is placed back at its original position.

There are a few obvious improvements: the delete/insertion code could be abstracted into a letf-bound function - but that seems like overkill. Another optimization is supporting triple quotes by using the looking-at function and passing it a regular expression that matches single or double quotes, in singles or triples, and replacing the match with the replacement character.

But the function works. And it could easily work for other modes with interchangeable quotes -- or even other paired expressions, like brackets. In fact, making it work with brackets is easier as the built-in command up-list (C-M-u) will go to the beginning of a balanced pair and forward-sexp will go to the end of the balanced pair at point.

Thanks to a little-known feature of Emacs's syntax parser you can make some simple assumptions about the text in a buffer and act on it in a structured manner.

Share

-1:-- Swapping quote symbols in Emacs with parse-partial-sexp (Post mickey)--L0--C0--August 26, 2014 03:18 PM

Chris Ball: Experimenting with Panamax

Disclosure: This blog post is part of the Panamax Template Contest.

In my blog post about the Dell C6100 server I’ve been using, I mentioned that I run a full LXC userland for each application I deploy, and that I’d like to try out Docker but that this setup is in conflict with Docker’s philosophy – a Docker container only runs one process, which makes it difficult to use Docker for anything requiring interaction between processes. Here’s an example: this blog is running WordPress with MySQL. So, with LXC I create a fresh Ubuntu container for the blog and run apt-get install wordpress and I’m up and running, but trying to use Docker would leave me with an “orchestration” problem – if I’m supposed to have a separate web server and database server, how will they figure out how to talk to each other?

If the two Docker services are being run on the same host, you can use docker --link, which runs one service under a given name and then makes it available to any service it’s linked to. For example, I could call a postgres container db and then run something like docker --name web --link db:db wordpress. The wordpress container receives environment variables giving connection information for the database host, which means that as long as you can modify your application to use environment variables when deciding which database host to connect to, you’re all set. (If the two docker services are being run on separate hosts, you have an “ambassador” problem to figure out.)

All of which is a long-winded way to say that Panamax is a new piece of open source software that attempts to ameliorate the pain of solving orchestration problems like this one, and I decided to try it out. It’s a web service that you run locally, and it promises a drag-and-drop interface for building out complex multi-tier Docker apps. Here’s what it looks like when pairing a postgres database with a web server running a Django app, WagtailCMS:

The technical setup of Panamax is interesting. It’s distributed as a CoreOS image which you run inside Vagrant and Virtualbox, and then your containers are launched from the CoreOS image. This means that Panamax has no system dependencies other than Vagrant and Virtualbox, so it’s easily usable on Windows, OS X, or any other environment that can’t run Docker directly.

Looking through the templates already created, I noticed an example of combining Rails and Postgres. I like Django, so I decided to give Django and Postgres a try. I found mbentley’s Ubuntu + nginx + uwsgi + Django docker image on the Docker Hub. Comparing it to the Rails and Postgres template on Panamax, the Django container lacks database support, but does have support for overlaying your own app into the container, which means you can do live-editing of your app.

I decided to see if I could combine the best parts of both templates to come up with a Panamax template for hosting arbitrary Django apps, which supports using an external database and offers live-editing.  I ended up creating a new Docker image, with the unwieldy name of cjbprime/ubuntu-django-uwsgi-nginx-live. This image is based on mbentley’s, but supports having a Django app passed in as an image, and will try to install its requirements. You can also link this image to a database server, and syncdb/migrate will be run when the image starts to set things up. If you need to create an admin user, you can do that inside a docker_run.sh file in your app directory.

After combining this new Docker image with a Postgres container, I’m very happy with how my django-with-postgres template turned out – I’m able to take an existing Django app, make minor changes using a text editor on my local machine to use environment variables for the database connection, start up the Panamax template, and watch as a database is created (if necessary), dependencies are installed, migrations are run, an admin user is created (if necessary), and the app is launched.  All without using a terminal window at any point in the process.

To show a concrete example, I also made a template that bundles the Wagtail Django CMS demo. It’s equivalent to just using my django-with-postgres container with the wagtaildemo code passed through to the live-editing overlay image (in /opt/django/app), and it brings up wagtaildemo with a postgres DB in a separate container. Here’s what that looks like:

Now that I’ve explained where I ended up, I should talk about how Panamax helped.  Panamax introduced me to Docker concepts (linking between containers, overlaying images) that I hadn’t used before because they seemed too painful, and helped me create something cool that I wouldn’t otherwise have attempted.  There were some frustrations, though.  First, the small stuff:

Underscores in container names

This one should have been in big bold letters at the top of the release notes, I think.  Check this out: unit names with _{a-f}{a-f} in them cause dbus to crash. This is amusing in retrospect, but was pretty inscrutable to debug, and perhaps made worse by the Panamax design: there’s a separate frontend web service and backend API, and when the backend API throws an error, it seems that the web interface doesn’t have access to any more detail on what went wrong. I’m lucky that someone on IRC volunteered the solution straight away.

The CoreOS Journal box occasionally stays black

Doing Docker development depends heavily on being able to see the logs of the running containers to work out why they aren’t coming up as you thought they would.  In Docker-land this is achieved with docker -f logs <cid>, but Panamax brings the logs in to the web interface: remember, the goal is to avoid having to look at the terminal at all.  But it doesn’t work sometimes.  There’s a panamax ssh command to ssh into the CoreOS host and run docker logs there, but that’s breaking the “fourth wall” of Panamax.

Progress bar when pulling Docker images

A minor change: it’d be great to be able to see progress when Panamax is pulling down a Docker image. There’s no indicator of progress, which made me think that something had hung or failed. Further, systemd complained about the app failing to start, when it just needed more time for the docker pull to complete.

Out of memory when starting a container

The CoreOS host allocates 1GB RAM for itself: that’s for the Panamax webapp (written in Rails), its API backend, and any containers you write and launch.  I had to increase this to 2GB while developing, by modifying ~/.panamax/.env:

export PMX_VM_MEMORY=2048

Sharing images between the local host and the container

I mentioned how Panamax uses a CoreOS host to run everything from, and how this drastically reduces the install dependencies.  There’s a significant downside to this design – I want to allow my local machine to share a filesystem and networking with my Docker container, but now there’s a CoreOS virtual machine in the way – I can’t directly connect from my laptop to the container running Django without hopping through the VM somehow. I want to connect to it for two different reasons:

  1. To have a direct TCP connection from my laptop to the database server, so that I can make database changes if necessary.
  2. To share a filesystem with a container so that I can test my changes live.

Panamax makes the first type of connection reasonably easy. There’s a VirtualBox command for doing port forwarding from the host through to the guest – the guest in this case is the CoreOS host. So we end up doing two stages of port forwarding: Docker forwards port 80 from the Django app out to port 8123 on the CoreOS host, and then VirtualBox forwards port 8123 on my laptop to port 8123 on the CoreOS host. Here’s the command to make it work:

VBoxManage controlvm panamax-vm natpf1 rule1,tcp,,8123,,8123

The filesystem sharing is much trickier – we need to share a consistent view of a single directory between three hosts: again, the laptop, the CoreOS host, and the Docker app. Vagrant has a solution to this, which is that it can NFS share a guest OS from the CoreOS host back to my laptop. That works like this, modifying ~/.vagrant.d/boxes/panamax-coreos-box-367/0/virtualbox/Vagrantfile:

  config.vm.network "private_network", ip: "192.168.50.4"
  config.vm.synced_folder "/home/cjb/djangoapp", "/home/core/django",
  id: "django", :nfs => true, :mount_options => ['nolock,vers=3,udp']

So, we tell Panamax to share /opt/django/app with the CoreOS host as /home/core/django, and then we tell Vagrant to share /home/cjb/djangoappon my laptop with the CoreOS host as /home/core/django over NFS. After `apt-get install nfs-kernel-server`, trying this leads to a weird error:

exportfs: /home/cjb/djangoapp does not support NFS export

This turns out to be because I’m running ecryptfs for filesystem encryption on my Ubuntu laptop, and nfs-kernel-server can’t export the encrypted FS. To work around it, I mounted a tmpfs for my Django app and used that instead. As far as I know, OS X and Windows don’t have this problem.

Summary

Panamax taught me a lot about Docker, and caused me to publish my first two images to the Docker registry, which is more than I expected to gain from trying it out. I’m not sure I’m the target audience – I don’t think I’d want to run production Docker apps under it on a headless server (at least until it’s more stable), which suggests that its main use is as an easy way to experiment with the development of containerized systems. But the friction introduced by the extra CoreOS host seems too great for it to be an awesome development platform for me. I think it’s a solvable problem – if the team can find a way to make the network port forwarding and the filesystem NFS sharing be automatic, rather than manual, and to work with ecryptfs on Ubuntu, it would make a massive difference.

I am impressed with the newfound ability to help someone launch a database-backed Django app without using any terminal commands, even if they’re on Windows and have no kind of dev environment, and would consider recommending Panamax for someone in that situation. Ultimately, maybe what I’ll get out of Panamax is a demystification of Docker’s orchestration concepts. That’s still a pretty useful experience to have.

-1:-- Experimenting with Panamax (Post cjb)--L0--C0--August 25, 2014 02:35 PM

Emacs Redux: A peek at Emacs 24.4: prettify-symbols-mode

Emacs 24.4 ships with a new minor mode called prettify-symbols-mode. Its purpose is to replace the standard text representation of various identifiers/symbols with a (arguably) more aesthetically pleasing representation (often a single unicode character would replace several ascii characters).

A classic example would be lambda from various Lisp dialects that many people prefer to replace with the greek letter λ (small lambda). prettify-symbols-mode allows you to achieve this by relying on a simple mapping expressed in the form of an alist that each major mode must initialize (prettify-symbols-alist). Simply put - major modes have to provide the configuration for prettify-symbols-mode.

Lisp modes do this via lisp--prettify-symbols-alist:

1
2
(defconst lisp--prettify-symbols-alist
  '(("lambda"  . ?λ)))

This means that out of the box only lambda will get replaced. You can, of course, add more mappings for different major modes:

1
2
3
(add-hook 'emacs-lisp-mode-hook
            (lambda ()
              (push '(">=" . ?) prettify-symbols-alist)))

Let’s see the mode in action. Consider this bit of Emacs Lisp code:

1
2
3
4
(lambda (x y)
  (if (>= x y)
      (something)
      (something-else)))

After you do M-x prettify-symbols-mode you’ll end up with:

1
2
3
4
(λ (x y)
  (if ( x y)
      (something)
    (something-else)))

To enable this for a particular mode use (add-hook 'some-mode-hook 'prettify-symbols-mode). If you’d like to enable it globally just add the following to your config:

1
(global-prettify-symbols-mode +1)

By the way, sky is the limit for symbol prettification. One fairly extreme example would be vim’s plugin haskell-conceal+ that goes to great lengths to bring proper mathematical notation to Haskell code. We can achieve more or less the same effect with prettify-symbols-mode, but one have to ask themselves where should we draw the border between tasteful and distasteful prettifications.

-1:-- A peek at Emacs 24.4: prettify-symbols-mode (Post)--L0--C0--August 25, 2014 01:30 PM

Tom Tromey: Emacs verus notification area, again

Ages and ages I wrote about letting Emacs code access the notification area.  I have more to say about it now, but first I want to bore you with some rambling thoughts and some history.

The “notification area” is also called the “status icon area” or the “systray” — it is a spot that holds some icons that are under control of various applications.

I was a fan of the notification area since it first showed up in Gnome.  I recognized it instantly as the thing I wanted that I hadn’t realized I wanted.

Now, as you know, the notification area has fallen on hard times.  It’s been removed in Gnome 3… I searched a bit for the rationale for this deletion, which as far as I can tell is just that some applications abused it, whatever that means; or that it was used inconsistently, which I think the web has conclusively proven is fine by users.  Coming from the Emacs perspective, where one can customize the somewhat-equivalent of the status area (see those recent posts on diminishing minor-mode lighters in the mode line…), and where a certain amount of per-mode idiosyncrasy is the norm, these seem like an inadequate reasons.

However, the reason doesn’t really matter.  I love the notification area!  When I moved more of my daily desktop use back into Emacs (the tides are strong but slow, and take years to come in or go out), I hooked Emacs up to it, and made it a part of my basic configuration.

It’s indispensable now.  What I particularly like about it is that it is both noticeable and unobtrusive — the former because I can have the icons blink on important events, and the latter because the icons don’t move around or obscure other windows.

Ok!  You should use it!  And I totally plan to tell you how, but first some boring history.

My original post relied on a hacked version of the Gnome zenity utility.  This turned out to be a real pain over time.  I had to rebuild it periodically, adding hacks (once removing chunks), etc.  Sharing it with others was hard.  And, for whatever reason, the patches in Gnome bugzilla were completely ignored.  Bah.

A bit later I wrote a big patch to Emacs to put all this into the core.  That patch was rejected, more or less.  Bah two.

Then even later I flirted with KDE for a bit.  Yes.  KDE had the nice idea to expose the notification area via dbus, and Emacs could talk dbus… so I did the obvious thing in elisp.  However the KDE notification area was pretty buggy and in the end I had to abandon it as well.

So, it was back to zenity… until this week, during my funemployment.  I rewrote my hacks in Python.  This was so easy I wish I’d done it years and years ago.

I’m not sure what the moral of this story is.  Maybe that my obsession is your gain.  Or maybe that I have trouble letting go.

Anyway, the result is here, on github, or in marmalade.  You’ll need Python and the new (introspection-based) Python Gtk interfaces.  This of course is no trouble to install.  The package includes the base status icon API, plus basic UIs for ERC and EMMS.  Try it out and let me know what you think.

-1:-- Emacs verus notification area, again (Post tom)--L0--C0--August 25, 2014 03:29 AM

Timo Geusch: Unique buffer names in Emacs

A common annoyance with Emacs when working on a code base that has duplicate file names is that the mode line tends to display the buffer names as “one.py:<1>”, “one.py:<2>” etc etc. That doesn’t help much with telling them apart and I find it confusing. I was introduced to the Uniquify library a while ago. […]
-1:-- Unique buffer names in Emacs (Post Timo Geusch)--L0--C0--August 23, 2014 10:58 PM

Tom Tromey: Another Mode Line Hack

While streamlining my mode line, I wrote another little mode-line feature that I thought of ages ago — using the background of the mode-line to indicate the current position in the buffer. I didn’t like this enough to use it, but I thought I’d post it since it was a fun hack.

First, make sure the current mode line is kept:

(defvar tromey-real-mode-line-format mode-line-format)

Now, make a little function that format the mode line using the standard rules and then applies a property depending on the current position in the buffer:

(defun tromey-compute-mode-line ()
  (let* ((width (frame-width))
     (line (substring 
        (concat (format-mode-line tromey-real-mode-line-format)
            (make-string width ? ))
        0 width)))
    ;; Quote "%"s.
    (setq line
      (mapconcat (lambda (c)
               (if (eq c ?%)
               "%%"
             ;; It's absurd that we must wrap this.
             (make-string 1 c)))
             line ""))

    (let ((start (window-start))
      (end (or (window-end) (point))))
      (add-face-text-property (round (* (/ (float start)
                       (point-max))
                    (length line)))
                  (round (* (/ (float end)
                       (point-max))
                    (length line)))
                  'region nil line))
    line))

We have to do this funny wrapping and “%”-quoting business here because the :eval form returns a mode line format — not just text — and because the otherwise appealing :propertize form doesn’t allow computations.

Also, I’ve never understood why mapconcat can’t handle a character result from the map function.  Anybody?

Now set this to be the mode line:

(setq-default mode-line-format '((:eval (tromey-compute-mode-line))))

The function above changes the background of the mode line corresponding to the current window’s start and end positions.  So, for example, here we are in the middle of a buffer that is bigger than the window:

Screenshot - 08222014 - 12:52:19 PM

I left this on for a bit but found it too distracting.  If you like it, use it. You might like to remove the mode-line-position stuff from the mode line, as it seems redundant with the visual display.

-1:-- Another Mode Line Hack (Post tom)--L0--C0--August 22, 2014 07:05 PM

Eric James Michael Ritz: Emacs: No More of My Own Code for PHP Mode

My apologies for the quiet month, as I’ve plans weighing on my mind. In the past I have asked for another Emacs Lisp developer to hopefully step up and offer to maintain PHP Mode. As of today I will no longer work on feature requests for the project. If developers continue to send me improvements and/or bug fixes then I will be happy to review and merge them as necessary. However, I feel like it is time to end my personal work on PHP Mode. I will not completely walk away and leave the project in a dead state until someone is willing to take over the role as maintainer. But from today forward I will no longer be an active contributor, instead simply merging contributions from the myriad of developers who have worked to improve PHP Mode to what it is today.

If you are a PHP programmer, use Emacs, and wish to improve the mode then please take a loot at the latest mode. If you have any questions feel free to ask any quenions.


-1:-- Emacs: No More of My Own Code for PHP Mode (Post ericjmritz)--L0--C0--August 22, 2014 04:54 PM

Jorgen Schäfer: Circe 1.4 released

I just released version 1.4 of Circe, a Client for IRC in Emacs.

The package is available from github and MELPA unstable, even though the latter will track further development changes, so use at your own risk.

Due to the sorry state of Emacs Lisp package archives, I am currently unable to do an actual release of Circe.

Changes

  • Non-blocking TLS connections. Circe now supports secure TLS connections to IRC servers without blocking during the connect. (Thanks to defanor for the patches!)
  • PEP (Python Enhancement Proposals) mentions are now buttonized by default, like RFCs and SRFIs.
  • URLs are now buttonized with the logic from the built-in thingatpt.el library.
  • Topic changes are highlighted correctly. Before, Circe would show diffs for topics one older than the last.
  • Circe will now try a random nick when the server says that the initial nick is erraneous. This could happen when the server set a collision-avoiding nick, and Circe tried to retain it after a reconnect.
  • The lui-logging module can now easily be enabled or disabled globally.
  • The new option circe-server-buffer-name can be used to influence the name of the server buffer, especially useful for people using the same bouncer for multiple networks. This can also be set on a per-network basis.
-1:-- Circe 1.4 released (Post Anonymous (noreply@blogger.com))--L0--C0--August 22, 2014 04:24 PM

Grant Rettke: prog-proc-mode

FWIW, I have started a `prog-proc-mode’, which is supposed to be a minor mode used in a programming mode and that makesthe link to an underlying comint mode.

– Stefan Monnier

It will make the link easier; every mode has to do it all themselves.

Via emacs.devel via ESS-help.

-1:-- prog-proc-mode (Post Grant)--L0--C0--August 22, 2014 01:34 PM

Alex Schroeder: Emacs and Dice

Somebody asked on Google+: What are the odds for rolling 2d10 and a d20 and getting higher on the 2d10?

(let ((a 0) (b 0) (n 0))
  (insert "2d10 vs. 1d20")
  (newline)
  (newline)
  (insert "     ")
  (dotimes (roll-1 10)
    (dotimes (roll-2 10)
      (insert (format " %5s"
		      (concat (number-to-string (1+ roll-1)) "+"
			      (number-to-string (1+ roll-2)))))))
  (newline)
  (dotimes (roll-3 20)
    (insert (format " %4d" (1+ roll-3)))
    (dotimes (roll-1 10)
      (dotimes (roll-2 10)
	(insert "     "
		(cond ((> (+ 2 roll-1 roll-2) (1+ roll-3))
		       (incf a)
		       "A")
		      ((= (+ 2 roll-1 roll-2) (1+ roll-3))
		       (incf n)
		       "-")
		      (t
		       (incf b)
		       "B")))))
    (newline))
  (newline)
  (let ((total (+ a b n)))
    (insert (format "A: %d / %d = %4.3f\n" a total (/ a 1.0 total)))
    (insert (format "B: %d / %d = %4.3f\n" b total (/ b 1.0 total)))
    (insert (format "-: %d / %d = %4.3f\n" n total (/ n 1.0 total))))
  (newline)
  (newline))

2d10 vs. 1d20

        1+1   1+2   1+3   1+4   1+5   1+6   1+7   1+8   1+9  1+10   2+1   2+2   2+3   2+4   2+5   2+6   2+7   2+8   2+9  2+10   3+1   3+2   3+3   3+4   3+5   3+6   3+7   3+8   3+9  3+10   4+1   4+2   4+3   4+4   4+5   4+6   4+7   4+8   4+9  4+10   5+1   5+2   5+3   5+4   5+5   5+6   5+7   5+8   5+9  5+10   6+1   6+2   6+3   6+4   6+5   6+6   6+7   6+8   6+9  6+10   7+1   7+2   7+3   7+4   7+5   7+6   7+7   7+8   7+9  7+10   8+1   8+2   8+3   8+4   8+5   8+6   8+7   8+8   8+9  8+10   9+1   9+2   9+3   9+4   9+5   9+6   9+7   9+8   9+9  9+10  10+1  10+2  10+3  10+4  10+5  10+6  10+7  10+8  10+9 10+10
    1     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    2     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    3     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    4     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    5     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    6     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    7     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    8     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
    9     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
   10     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A     A
   11     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A     -     A     A     A     A     A     A     A     A     A
   12     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A     B     -     A     A     A     A     A     A     A     A
   13     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A     B     B     -     A     A     A     A     A     A     A
   14     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A     B     B     B     -     A     A     A     A     A     A
   15     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A     B     B     B     B     -     A     A     A     A     A
   16     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A     B     B     B     B     B     -     A     A     A     A
   17     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A     B     B     B     B     B     B     -     A     A     A
   18     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A     B     B     B     B     B     B     B     -     A     A
   19     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -     B     B     B     B     B     B     B     B     -     A
   20     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     B     -

A: 1000 / 2000 = 0.500
B: 900 / 2000 = 0.450
-: 100 / 2000 = 0.050

So, the odds of rolling higher on 2d10 is 50%.

(Does anybody have some actual math on how to do this?)

Tags: RSS RSS

-1:-- Emacs and Dice (Post)--L0--C0--August 22, 2014 07:25 AM

M-x all-things-emacs: The Editor of a Lifetime

Perry Metzger has been using Emacs as his text editor since early September, 1983 — nearly 31 years. Over much of that time, it has also been his primary way to read email, compile programs, and perform a variety of other tasks.
Why would anyone use a single program for that long? This talk is partially intended to answer that question.
Emacs remains one of the most important user interfaces (and text editors) for computer professionals almost 40 years after it was created. The talk is intended to be part history, part philosophy, and part speculation on the future. It will also teach Emacs fans how to explain to their skeptical friends why it is still a good idea to learn a tool from the terminal era that requires memorization of dozens of control sequences in an age of GUIs and smart phones.
-1:-- The Editor of a Lifetime (Post Ryan McGeary)--L0--C0--August 21, 2014 04:47 PM

Got Emacs?: Emacs fourth pretest released

Missed the announcement but the fourth pretest is now available for testing.
-1:-- Emacs fourth pretest released (Post sivaram (noreply@blogger.com))--L0--C0--August 20, 2014 05:55 PM

Julien Danjou: Tracking OpenStack contributions in GitHub

I've switched my Git repositories to GitHub recently, and started to watch my contributions statistics, which were very low considering I spend my days hacking on open source software, especially OpenStack.

OpenStack hosts its Git repositories on its own infrastructure at git.openstack.org, but also mirrors them on GitHub. Logically, I was expecting GitHub to track my commits there too, as I'm using the same email address everywhere.

It turns out that it was not the case, and the help page about that on GitHub describes the rule in place to compute statistics. Indeed, according to GitHub, I had no relations to the OpenStack repositories, as I never forked them nor opened a pull request on them (OpenStack uses Gerrit).

Starring a repository is enough to build a relationship between a user and a repository, so this is was the only thing needed to inform GitHub that I have contributed to those repositories. Considering OpenStack has hundreds of repositories, I decided to star them all by using a small Python script using pygithub.

And voilà, my statistics are now including all my contributions to OpenStack!

-1:-- Tracking OpenStack contributions in GitHub (Post Julien Danjou)--L0--C0--August 19, 2014 05:00 PM

Julien Danjou: OpenStack Ceilometer and the Gnocchi experiment

A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning.

In this article, I would like to relate what I've been doing with other Ceilometer developers in the last 5 months. I've lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it's largely time to take a break and talk about it.

Ceilometer early design

For the last years, Ceilometer didn't change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called samples. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster…) constructs and emits a sample headed for the storage component that we call the collector.

This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support.

The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want.

The scalability issue

We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each sample generated by Ceilometer and is stored as is). That basically means that the sample data structure is stored in most drivers in just one table or collection, in order to be able to scan them at once, and there's no good "perfect" sharding solution, making data storage scalability painful.

It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are O(n) where n is the number of samples recorded (see big O notation if you're unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable.

During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.

Thinking outside the box

Unfortunately, the PTL job doesn't leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn't able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box.

When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics.

Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built into oslo.messaging.

Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth usage, the number of images that Glance is storing for you, etc… These are not events, since nothing is happening. These are facts, states we need to meter.

Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right.

I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there's still no silver bullet – this made it a good challenge.

The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API – as we do in all OpenStack services. This should cover the metric storage pretty well.

Cooking Gnocchi

A cloud of gnocchis!

At the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a "pasta" name, right?

The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process.

The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I've named these things "entities", as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn't care and should be agnostic in this regard.

One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the RRDtool have been doing for a long time on the fly, and I decided it was a good trail to follow.

I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both.

The first driver written to achieve that and store those metrics inside Gnocchi was based on whisper. Whisper is the file format used to store metrics for the Graphite project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable.

Storing metrics for each entities in a different whisper file and putting them in Swift turned out to have a fantastic algorithm complexity: it was O(1). Indeed, the complexity needed to store and retrieve metrics doesn't depends on the number of metrics you have nor on the number of things you are metering. Which is already a huge win compared to the current Ceilometer collector design.

However, it turned out that whisper has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (time.time()). I've started to hack on that in my own fork, but… then everything broke. The whisper project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform whisper into the time-series format I wanted, without being sure I wasn't going to break everything (remember, no test coverage).

I decided to take a break and look into alternatives, and stumbled upon Pandas, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it carbonara (a wink to both the Carbon project and pasta, how clever!). The code is quite small (a third of whisper's, 200 SLOC vs 600 SLOC), does not have many of the whisper limitations and… it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers.

Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It's a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a InfluxDB driver, working closely with the upstream developer of that database. Dina Belova started to write an OpenTSDB driver. This helps to make sure the API is designed directly in the right way.

Handling resources

Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.

Relationship of entities and resources

Once this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now.

Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network.

All of these requirements led to to the design of what's called the indexer. The indexer is responsible for indexing entities, resources, and link them together. The initial implementation is based on SQLAlchemy and should be pretty efficient. It's easy enough to index the most requested attributes (columns), and they are also correctly typed.

We plan to establish a model for all known OpenStack resources (instances, volumes, networks, …) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It'd be up to the users to store extra attributes.

Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.

The indexer classes and their relations

REST API

All of this is exported via a REST API that was partially designed and documented in the Gnocchi specification in the Ceilometer repository; though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer.

The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.

Macroscopic view of the Gnocchi architecture

Roadmap & Ceilometer integration

All of this plan has been exposed and discussed with the Ceilometer team during the last OpenStack summit in Atlanta in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue.

It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to.

Early benchmarks

Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July.

The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.

For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of O(1), and not O(n) anymore.

Next steps

Clément drawing the logo

While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now.

The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We'll discuss how to integrate it in the project in a more permanent and strong manner probably during the OpenStack Summit for Kilo that will take place next November in Paris.

-1:-- OpenStack Ceilometer and the Gnocchi experiment (Post Julien Danjou)--L0--C0--August 18, 2014 03:00 PM

Mickey Petersen: dirswitch.el: a fish shell-style directory switcher for shell-mode

A friend of mine showed me Fish Shell, a shell replacement for Mac and Linux. One of its coolest features was a “quick directory switcher” that lets you jump to directories you’ve previously visited in that session.

Feeling left out I decided to write dirswitch.el, a directory switcher for M-x shell, Emacs’s built-in shell mode (see Running shells in Emacs: an Overview.)

Like Shell’s M-r history search functionality (which works much like isearch) dirswitch.el will record directories you visit and let you rapidly switch between them by pressing C-M-n and C-M-p.

It’s still a rough prototype but it’s a hallmark example of what a few hours of Emacs hacking can accomplish.

You can get it from my Github and it’ll probably appear in a package manager near you soon.

Share

-1:-- dirswitch.el: a fish shell-style directory switcher for shell-mode (Post mickey)--L0--C0--August 17, 2014 05:22 PM

Alex Schroeder: Gnus Mystery Emails

These days I have a Raspberry Pi running Dovecot running as my IMAP Server. I connect to this IMAP Server from my iPhone using Apple’s Mail and from my Laptop using Gnus. For a while now I have noticed that the two don’t list the same mails in my INBOX. In Gnus, I enter the group using C-u RET or I use / o to insert old articles and I see 14 mails. When I enter the Server buffer from the Group buffer using ^ and switch to the INBOX, I see 72 mails. What could be the reason for this?

Tags: RSS RSS

-1:-- Gnus Mystery Emails (Post)--L0--C0--August 14, 2014 07:54 AM

Bryan Murdock: Free Verilog Simulators

At DVCon 2013 I asked JL Gray's panel if we would ever have Free tools, like the software world. None of panelists seemed to think so, one of the panelists, a Mentor employee, scoffed, "you get what you pay for with free tools." Never mind that their (and Cadence's and Synopsys's) products are very likely developed with tools that contain millions of lines of Free software.

So, to work towards answering my own question, I spent a little time and looked for Free/Open Source verilog simulators. Here's what I found:

Icarus Verilog

GPL Cver

PVSim Verilog Simulator

VeriWell Verilog Simulator

I have personally used Icarus and Cver before, but not very extensively.  They were usable and seemed pretty complete, for Verilog.  None of the above claim any support of SystemVerilog except for Icarus.  The Icarus developer at one point expressed abhorrence at SystemVerilog but it seems support for some parts of the language have been added.

PVSim and VeriWell were new to me.  I'll give them a try, hopefully soon, and post more information.

Another one that should be mentioned is Verilator.  I have downloaded and played with this one too.  It only supports synthesizable Verilog, so have fun writting a testbench.  I think the intent is for you to write your testbench in C++, so if you like that idea then this could be a good one to try too.

Did I miss any?

UPDATE: All of these (except Verilator) plus a host of other free EDA tools are available to easily try out at EDA Playground. Go there now, it's a fun place to play.

2ND UPDATE: As mentioned in the comments below, there are two other ways to try Icarus online: www.compileonline.com and www.iverilog.com.

-1:-- Free Verilog Simulators (Post Bryan (noreply@blogger.com))--L0--C0--August 12, 2014 02:31 PM

Bozhidar Batsov: The State of Some Emacs Packages for Clojure Development

There are quite a few packages in the “official” clojure-emacs GitHub organization, but many of them have been deprecated recently with the release of CIDER 0.7. Unfortunately not everyone is aware of this yet and I often see tickets related to those deprecated projects. In this short post I’ll outline the deprecations and provide a bit of background for them.

clojure-test-mode

The venerable clojure-test-mode was deprecated in favor of cider-test (which is bundled with CIDER 0.7). clojure-test-mode featured quite a lot of inlined Clojure code, which made the package very hard to maintain and reworking it to use nREPL middleware was a no-brainer for us. clojure-test-mode will be removed from the clojure-mode repo at some point. It also interferes with CIDER’s initialization, so you’re strongly encouraged to get rid of it.

Down the road we might extend cider-test to support other test frameworks as well (which should be feasible with different middleware providing the same interface).

company-cider

company-cider was deprecated, because company-mode integration was added to CIDER itself (making company-mode the officially supported and recommended completion library).

ac-nrepl

ac-nrepl has been superseded by ac-cider. ac-cider has a simpler codebase and leverages the compliment-based completion introduced in CIDER 0.7. We’ll probably remove ac-nrepl at some point in the future to avoid the confusion between the two.

cider-inspect

cider-inspect was absorbed into CIDER 0.7.

cider-tracing

cider-tracing was superseded by middleware-based tracing support integrated in CIDER 0.7.

Epilogue

Those deprecations are also mentioned in the documentation of the respective packages, but I feel it’s nice to have them listed together in a single document. Most of the packages will also emit load-time deprecation warnings.

-1:-- The State of Some Emacs Packages for Clojure Development (Post)--L0--C0--August 12, 2014 08:29 AM

Jorgen Schäfer: Elpy 1.5.1 released

I just released version 1.5.1 of Elpy, the Emacs Python Development Environment. This is a bug fix release.

Elpy is an Emacs package to bring powerful Python editing to Emacs. It combines a number of other packages, both written in Emacs Lisp as well as Python.

Quick Installation

Evaluate this:

(require 'package)
(add-to-list 'package-archives
'("elpy" .
"http://jorgenschaefer.github.io/packages/"))

Then run M-x package-install RET elpy RET.

Finally, run the following (and add them to your .emacs):

(package-initialize)
(elpy-enable)

Changes in 1.5.1

  • Fix a bug where company-mode might get confused about the current backend, leading to an error about Symbol’s function definition is void: nil
  • Fix Rope so it won’t search the whole project directory. This was an intended feature in v1.5 which did not work originally.
  • Use yas-text instead of text in snippets for compatibility with the unreleased yasnippet from MELPA (thanks to Daniel Wu!)
-1:-- Elpy 1.5.1 released (Post Anonymous (noreply@blogger.com))--L0--C0--August 10, 2014 07:23 AM

Bozhidar Batsov: CIDER 0.7

CIDER 0.7 is finally out and it’s an epic release! It’s without a doubt the most important release since the inception of the project about two years ago and it’s the biggest one in terms of features and code changes.

The release is special for a number of reasons. Allow me to quickly enumerate though them.

Middleware-based

One of the huge problems we’ve had so far was that a lot of functionality that was present in SLIME + swank-clojure was missing in CIDER. For many people the transition between the two didn’t really feel like an upgrade (although it was advertised as such) – after all they lost things like inspection, tracing, apropos, caller cross-reference, the debugger, etc.

The reason this happened was CIDER’s initial approach of implementing features by inlining Clojure code within the Emacs Lisp code and evaluating this code in a dedicated nREPL session (called the tooling session), to avoid contaminating the results in the “primary” eval session. This approach had one upside (you didn’t need any extra deps (middleware) to run CIDER) and one huge downside (it’s impossible to maintain non-trivial inlined code; not to mention it’s not very practical). This meant that pretty much all third party libs were out of the equation and pretty much every advanced feature.

This problem was initially address by ritz, which provided some extra functionality built on top of extra nREPL middleware. As it was a separate project it was hard to be kept in sync with the flurry of changes in CIDER and was abandoned at some point. While ritz failed it had the right approach and it served as the principle inspiration for CIDER 0.7.

In CIDER 0.6 we introduced optional nREPL middleware for some operations (like completion, error reporting, var info) and in 0.7 the middleware stack was greatly extended, improved and promoted to a mandatory CIDER component. This has one downside (you’ll have to install it to leverage all of CIDER’s power) and a several upsides:

  • A lot of the heavy lifting is now done in pure Clojure code (see cider-nrepl) and it’s much easier to implement complex features now (not to mention – this code is much easier to maintain). This also means that it’s much easier for Clojure programmers to contribute to CIDER as a lot of functionality is just lots of Clojure code and very little Emacs Lisp code. I’m reasonably sure I’m one of the very few CIDER users who knows more Emacs Lisp than Clojure, so I consider this a big win.

  • It’s now easy to provide pretty similar level of support for both Clojure and ClojureScript as we can reconcile the differences between them in our middleware.

  • It’s easier to keep the code backwards compatible (which is a nightmare for inlined code).

  • There are no implicit dependencies in the system (unlike before).

Consider auto-completion – this feature was implemented in terms of evaluating some clojure-complete code with the assumption that clojure-complete is available in the environment you were using as CIDER assumed you had started the nREPL server using lein repl. This would fire up a REPL-y REPL and clojure-complete is a REPL-y dependency. Not everyone uses lein repl, though and REPL-y can always switch to another completion library in the future.

If you connected CIDER to an embedded nREPL server, you’d be greeted by a missing class error, as most apps don’t normally depend on clojure-complete. You’d be puzzled for a while, but eventually you’ll realize what the problem is.

Now we’re free to explicitly specify our deps and pick the best libraries for the job (as opposed to those that are available) – you’ll quickly notice how smarter auto-completion is now on Clojure, because we’re internally using the newer, faster and more feature-rich compliment library (note that we’re using a different library for ClojureScript completion). * Other projects can leverage some of our middleware – somewhat amusingly for Emacs users, vim-fireplace is using cider-nrepl as well.

At this point we’ve removed pretty much all inlined code (except some pretty-printing code) and that has yielded much improved eldoc, macroexpansion, documentation viewing (cider-doc will now display Javadoc in Emacs!!!), source browsing, etc.

We’ve also started bringing back some features we loved in SLIME, but were missing so far – the inspector, apropos and tracing are back. We’re now working on bringing back function call cross-referencing and debugging as well.

Note that CIDER will still work if you connect to an nREPL server that’s not using CIDER’s middleware. In this case you’ll get a warning and a pretty limited feature-set – source file loading, code evaluation, pretty-printing and error highlighting.

cider-test

clojure-test-mode (which was more or less abandoned in terms of maintenance) finally has a successor in CIDER itself. cider-test provides more or less the same functionality, but is implemented in terms of nREPL middleware and is a more robust solution. As it’s part of CIDER it cannot ever be out-of-sync with CIDER as clojure-test-mode has often been lately.

cider-test is a little rough around the edges, but I’m fairly sure it has bright future ahead. Use it, love it, hate it and send us your feedback! We’d love to hear it.

P.S. We might extend this with support for other frameworks like midje and expectations, although that’s not high on our priority list.

Grimoire support

In addition to built-in Clojure & Javadoc you can now peruse the extended documentation provided by Grimoire from the comfort of your beloved editor. No more browser interruptions just to get a few usage examples of some function! C-c C-d g for the win!

Adding some extended Grimoire integration is definitely on the roadmap.

Increased bus factor

One of the biggest problems of the project so far was that fairly few people were involved with it. At one point it was mostly Tim, Hugo and me. At another it was mostly me. The bus factor was dangerously close to 1, which always worried me. Recently, however, a lot of people have been helping quite actively, which makes me more optimistic about the future. As much as I love CIDER I don’t want it to depend on one extremely busy and very clumsy person (each time I go hiking there’s a serious chance I’ll fall of a cliff or something).

I’d like to thank everyone for your wonderful contributions and single out a bunch of people for some outstanding work done by them:

  • Gary Trakhman is the one responsible for the good ClojureScript support we now boast. Fantastic work, Gary! You have a big thanks from me!
  • Jeff Valk did some mighty fine work on the var info retrieval, source navigation, documentation display and single-handedly implemented cider-test and cider-apropos. We all owe Jeff a huge thanks!
  • Hugo Duncan who constantly contributed patches, bug reports and ideas. His nrepl-ritz project provided a lot of inspiration for some of the existing middleware.
  • Ian Eslick contributed the new inspector.
  • Alexander Yakushev made so many improvements to his awesome compliment library for the needs of CIDER. Did you notice that you now get completion suggestions for locals? How amazing is that!
  • Dmitry Gutov implemented native support in CIDER for company-mode.

The Road Before Us

Obviously our work is far from over – we’re still lacking some crucial features (most notably a debugger) and a lot of code needs polish. This will obviously take some time and a lot of work to get right, but I’m confident we’ll get there.

I’d also want us to work a bit in the area of documentation – a manual, a cheatsheet, etc. Have a look at the issue tracker if you’d like to help out – we definitely need all the help we can get.

The 0.7 release required a massive amount of work and we spent more that 3 months to get it to a shippable state. With the bulk of the work behind us I hope we’ll be able to deliver new releases more frequently – on a monthly (or bi-monthly) basis.

I’d love to be able to raise a lot of money via crowd-funding and work on CIDER for an year or get hired by some company to work full-time on it, but that’s not going to happen. For one reason or another people rarely get excited about dev tools, so all of us have to work together to make CIDER the ultimate Clojure(Script) development environment.

If I knew how much work I’d have to do when I assumed the maintenance of CIDER exactly one year ago I might not have done it. Maintaining a project that’s pivotal to an entire community (I recall some article mentioning about half the Clojure devs were using it) is both a lot of work and a lot of stress and pressure. That said, it’s probably my favourite OSS project and I enjoy working on it immensely. If you don’t have the time to help out with code or docs you can still support my work on CIDER (and all my other projects) via gittip.

Support via Gittip

For all the gory details regarding new features and changes in CIDER 0.7 – take a look at the changelog.

That’s all from me, folks! Use CIDER, drink cider and prosper!

-1:-- CIDER 0.7 (Post)--L0--C0--August 05, 2014 12:19 PM

John Sullivan: No such pipe, or this pipe has been deleted

This data comes from pipes.yahoo.com but the Pipe does not exist or has been deleted.
-1:-- No such pipe, or this pipe has been deleted (Post)--L0--C0--August 03, 2014 02:58 PM

Flickr tag 'emacs': 2014-08-03_032527_1366x768_scrot

Thiago Perrotta posted a photo:

2014-08-03_032527_1366x768_scrot

Example of how to use org mode to manage a project.

-1:-- 2014-08-03_032527_1366x768_scrot (Post Thiago Perrotta (nobody@flickr.com))--L0--C0--August 03, 2014 06:27 AM

Flickr tag 'emacs': enviando-y-recibiendo

El Diego Efe posted a photo:

enviando-y-recibiendo

-1:-- enviando-y-recibiendo (Post El Diego Efe (nobody@flickr.com))--L0--C0--July 31, 2014 08:26 PM

Eric James Michael Ritz: GNU Emacs: Competing With Other Editors—Should It?

Today while browsing the Emacs Reddit I came across this interesting thread. The sensationlist title contains the following quote from co-maintainer Stefan Monnier:

We’re not in the business of competing.

He is referring to competing with other editors and IDEs. Today I want to frame his comment in the proper context and discuss whether GNU Emacs should compete with the likes of Visual Studio and Eclipse.

The Context

Stefen’s comments come from a thread discussing the policy for packages in ELPA, the official Emacs package repository. In the emacs-24 branch of the official repository, the Python mode has going through various non-bug–fix changes. Some people believe this should not happen. Changes to language-specific modes should adhere to the same feature freeze as everything else. Other people believe that changes to such modes should always be welcome, assuming that don’t break anything. One argument for this is that it helps Emacs stay up to do with editors that pump out new features, and this would help Emacs keep up.

And here came the, “We’re not in the business of competing comment.”

The Benefits of Not Competing

GNU Emacs has long done its own thing. Any long-time user will tell you that. This attitude allows the Emacs developers to focus on what interests them without being beholden to the trends and practices of editors such as Visual Studio. Granted—a lot of Emacs Lisp developers will simply ‘steal’ those features be re-writing them in Emacs Lisp, but that’s beside my point.

Decades of this attitude has created an atmosphere around Emacs where newcomers are expected to conform to the standards of Emacs, or to move on to something else. “Why isn’t Control+S ‘save’ instead of starting a search?” I’ve seen these questions a lot. Emacs fans tend to trot out three responses:

  1. You can bind the keys to whatever you want. (Non-obvious to most on how to do that.)

  2. Emacs was that way long before Control+S because the standard for save. (Clinging to the past in an effort to avoid change.)

  3. You’ll get used to it soon enough. (The weakest of all rebuttals.)

It’s worth mentioning CUA Mode helps these issues. But then you have the long-term users who seem to view CUA Mode with the same disdain that the Brazilian national football teams has when it comes to guarding their goal against Germans.

Long Term Objectives

I saw the following quote in the discussion linked above. It in no way reflects the Free Software Foundation. That said, this is not the first time I’ve seen this?

They do not want to succeed in any of these criteria. Users, developers, and donations are nice to have, but simply not a priority of GNU and the FSF.

The only thing they really want to succeed at is freedom, and by their measures Emacs already succeeded in this aspect, being a free software project under a strong copyleft license, owned by a non-profit foundation.

And that freedom does not fade or decrease if Emacs fail to attract users, developers or donations. The license ensures that Emacs is going to remain free as long as our legal system persists, even if no one was left using or developing it.

The Hell…? As another poster succintly put it, “Freedom without people, that’s an interesting goal.”

The Free Software Foundation (FSF) cannot have an attitude like this. They must realize that they compete with other editors whether they want to or not. Many programmers seem to make their choice in editors after trying each for about fifteen to thirty minutes in my experience. Emacs should do whatever it can to rope in new users within that time frame. It’s esoteric behavior that made sense decades ago no longer stands against todays standards.

Now then, the power to customize Emacs is second-to-none. But there is no reason to expect a new user approach Emacs to even realize that. Even worse, some users feel like such new users are hurting the community. They would rather have Emacs as an inclusive club for people who have invested the effort in its arcane arts.

Conclusion

How to change this community culture? I will write my thoughts in the future. But I admit to having no simple answers. If you have any thoughts, ideas, or disagreements then please leave them in the comments.


-1:-- GNU Emacs: Competing With Other Editors—Should It? (Post ericjmritz)--L0--C0--July 26, 2014 11:28 PM

Grant Rettke: R 3.1.1 out and it fixes the package manager function

Via gog.

-1:-- R 3.1.1 out and it fixes the package manager function (Post Grant)--L0--C0--July 26, 2014 08:57 PM

Timo Geusch: Sprinkler controller upgrade part III – setting it up

Putting the OpenSprinkler and Raspberry Pi together was easy, getting them to run showed my inexperience when it comes to playing with hardware. The overall install went pretty smoothly and the documentation is good and easy to follow so I’m not going to ramble on about it for very long, but just throw up some […]
-1:-- Sprinkler controller upgrade part III – setting it up (Post Timo Geusch)--L0--C0--July 25, 2014 02:39 PM

Ivan Kanis: Learning Lisp is like learning ancient Greek

Learning Lisp is like learning ancient Greek; although it is alien, it provides insight into modern computer languages.

-1:-- Learning Lisp is like learning ancient Greek (Post Ivan Kanis)--L0--C0--July 21, 2014 12:00 AM