Reconsidering Our Stack

[aminb]

Hey everyone,

TLDR

The other day @samer and I had a brief chat about the technology (language/framework) used for EmacsConf’s front page, emacsconf2015.org.

As of now, the front page is a rails app (repo), however we are thinking maybe it’s a good idea to reconsider this choice, that whether using rails is necessary. We have discourse as a discussion forum / mailing list, gitit (wiki) for delivering content and a gogs instance for hosting our git repos. Also, there’s a good chance that in future we’d host a MediaGoblin instance for our “media” needs.

With that said, and with all these apps/tools available to us, do we still want to use rails for our front page, or we could very well use static pages (or a bit fancier, static page generators like Jekyll or Pelican) ?
Or, if we decide that we indeed need a proper web app, how about using Python and Django instead of Ruby on Rails?

[phr]

The front page looks to me like a static html page with some links on it, which is great. Why is any kind of app needed?

I think Django is kind of old-school for Python these days. My buddy who does Python web stuff prefers Flask.

Or if we want to be Emacsical(?) about it, how about org-mode M-x publish? There is also this, based on org-mode: http://ahungry.com/blog/2013-04-01-blogging-with-org-mode.html

I do like the idea of a blog-format front page so people would know where to look for updates and announcements.

If you’re into functional programming and want to see something cool and pointy-headed, check out Ur/web (www.impredicative.com/ur/). That’s only if you want a science project though.

Regarding the rest of the stack, I don’t want to diss too much on anyone’s favorite software, but I notice we’re using Discourse primarily as a bug and issue tracker, which makes me wonder if we should use an actual one. Discourse and Gitit are also both pretty bloated, and we don’t seem to be using gitit at all so far. I do like Gogs from what I’ve seen of it so far (even if we’re not using it yet either).

There are various things not to like about Fossil (www.fossil-scm.org) but it sure does a lot for its size (DVCS, wiki, and bug tracker in a single C program with footprint under 2MB). Samer and I considered it as an alternative to Gitit when we were discussing what wiki to use. It might be interesting to try porting some of its features into Gogs.

I like the idea of keeping the software stack small so that the site can be left up indefinitely without spending a lot on recurring hosting costs. I’ve been using LES-style servers (www.lowendspirit.com) for various things and for a low traffic site like ours, if we can’t run on one it feels to me like we’re doing something wrong. They start at under 2 dollars per YEAR (64mb nanovz.com, 1.50 euro) though the normal (128mb) ones are about 2x that amount.

[samer]

Regarding gitit: we will need it when we create “resource” pages. We’re not there yet, though.

Also, I’m okay with going with a static site for the front page. I think the next biggest piece we need is some sort of events sign up and calendar, as well as a blog. If we can do that in rails easily and integrate them w/ the front page, that seems like a good thing to do.

I would be open to using a “real” bug/task tracker. The issue with that is most bug trackers are not very good. I’ve been keeping track of my tasks offline, and it would be nice if we had a place online to quickly view what everyone is working on. Any ideas? Can we use gitit for this?

[phr]

I think it’s fine to use gitit for tasks and stuff. Wikipedia organizes far more complicated efforts than ours, entirely on wiki pages.

You are right about bug trackers–I’ve never seen one that didn’t suck. One can always hope, though.

I personally find Rails kind of distasteful (too much boilerplate and bureaucracy) but if you want to use it, what the hey ;-).

Unless we want to keep signups private, I think it’s ok to use the wiki for signups. It’s not like we’re doing events with 1000’s of people or selling tickets, that would need fancier approaches. Similarly I think it’s fine to have a purely static blog, since we can use Discourse if we want comment threads for blog posts.

[aminb]

The front page looks to me like a static html page with some links on it, which is great. Why is any kind of app needed?

Well, basically, for now, it is a static page, running as a rails app. Hence, my question (:

I think Django is kind of old-school for Python these days. My buddy who does Python web stuff prefers Flask.

This was on HN front page a few days ago. It really depends on the scenario. I think in our case going with Django would be easier and we’d have a working thing sooner, than if we’re to go with Flask.

Or if we want to be Emacsical(?) about it, how about org-mode M-x publish? There is also this, based on org-mode: http://ahungry.com/blog/2013-04-01-blogging-with-org-mode.html

Technically we could (that’s how http://git.emacsconf2015.org/emacsconf/emacsconf2015 is done, right?) but I don’t think it’ll be enough for our needs.

I do like the idea of a blog-format front page so people would know where to look for updates and announcements.

From personal experience, static site generators (e.g. Jekyll, Hakyll, Pelican) are best for this, if we decide not to use a web framework.

If you’re into functional programming and want to see something cool and pointy-headed, check out Ur/web (www.impredicative.com/ur/). That’s only if you want a science project though.

Personally I don’t have the time / dedication right now, but that just went in my watchlist.

Regarding the rest of the stack, I don’t want to diss too much on anyone’s favorite software, but I notice we’re using Discourse primarily as a bug and issue tracker, which makes me wonder if we should use an actual one. Discourse and Gitit are also both pretty bloated, and we don’t seem to be using gitit at all so far. I do like Gogs from what I’ve seen of it so far (even if we’re not using it yet either).

Yeah I think although Discourse looks pretty nice, it is somewhat heavy. Gogs is pretty nice and lightweight, and it’s actually being used right now. But the one sad thing about it is that it doesn’t have pull requests implemented, yet.

There are various things not to like about Fossil (www.fossil-scm.org) but it sure does a lot for its size (DVCS, wiki, and bug tracker in a single C program with footprint under 2MB). Samer and I considered it as an alternative to Gitit when we were discussing what wiki to use. It might be interesting to try porting some of its features into Gogs.

Fossil looks really interesting and it could be a real source of inspiration.

I like the idea of keeping the software stack small so that the site can be left up indefinitely without spending a lot on recurring hosting costs. I’ve been using LES-style servers (www.lowendspirit.com) for various things and for a low traffic site like ours, if we can’t run on one it feels to me like we’re doing something wrong. They start at under 2 dollars per YEAR (64mb nanovz.com, 1.50 euro) though the normal (128mb) ones are about 2x that amount.

Wow that looks pretty interesting!

[aminb]

@phr @samer guys you have to check this out: http://lug.ncsu.edu/

It’s a Django app. Looks really nice and seems like they’ve implemented a good number of features already (including a simple wiki).

They’ve released the source code as free software (GPLv3), we could just fork it and start using it right away. Also, since I’m quite comfortable writing Python, we could keep adding the features that we need.

I think this is just awesome.

What do you say?

[phr]

Thanks for that django/flask comparison, which is interesting. I’ve never used either of them but if it were me I’d use Flask based on my friend’s recommendation and so I could use his code and ask him things. YMMV. Why would we want a framework anyway though? What kind of code or application would we want to write (just event signups/calendar)? Or are there existing modules (Rails gems, etc.) that you think we’d want to use? If we’re writing our own stuff, would it be more in the Emacs spirit to use something written in Lisp, even if it’s not Emacs Lisp?

Oh that’s unfortunate about Gogs not having pull requests. I wonder how hard it would be to add them.

One of the things I like about Fossil is how fast its home page always loads, even though it’s running on a VPS.

http://lug.ncsu.edu/ could be useful, though for something like Emacsconf I’m just not that excited by the idea of yet another Django/Flask/Rails site, and I’m always infuriated by site designs that eat vertical screen space with a fixed horizontal navigation bar. I don’t want to stand in anyone’s way about anything though. I do know Python pretty well so if this is just a “get things done” task, then that Django app is probably a good choice. If it’s also “be way cool” then I prefer things that are more hackerish (Rrix has me interested in Lnode even though it sounds terrible) and/or avant-garde (ur/web being an extreme example).

I wonder if there’s a framework for the nginx lua/redis modules, which apparently are blazing fast, especially with Luajit (Cloudflare uses it). And Lua can be fronted with Purescript (purescript.org) for that extra nerdery.

Amin, http://git.emacsconf2015.org/emacsconf/emacsconf2015 is just an org file slapped onto a git page expecting markdown, so it looks awful. The ahungry blog looks like this: http://ahungry.com/blog/
I don’t know if there’s a way to get it to include the post texts on the front page. of course we could add that if it’s not there now.

[samer]

Just to reorient ourselves: the reason why I wanted to go with rails
because I don’t want to write the entire stack myself – there should be
pieces that already exist (like an events & calendar system) that we can
just plug in.

I’m not married to rails, but I want to see if we can do that with
rails before we explore other options. So in this case, I’m prioritizing
“getting it done” over having cool tech

s/rails/django/g if we decide to go with Django instead.

Of course, we can reimplement this stuff in a “fun” way after we get it
working. I’ve just been a part of too many “let’s do boring activity X
in fun technology Y” projects that never got finished because X is
still boring, even if you’re doing X in Haskell. So I think we should
take the pragmatic approach here.

So, before we move forward discussing these technologies, let’s really
hammer out what our goals are for the site.

I’ve made a wiki page for it here:
http://wiki.emacsconf2015.org/Planning

We should still discuss things on Discourse, but we should use the wiki
to keep track of what we decide on.

Thoughts?

I think it’s fine to use gitit for tasks and stuff. Wikipedia
organizes far more complicated efforts than ours, entirely on wiki
pages.

Okay, let’s start using Gitit for organizing tasks and stuff.

Unless we want to keep signups private, I think it’s ok to use the
wiki for signups.

I think signups needs to be private because we still need to collect
emails. Otherwise, the wiki would work for that, too.

guys you have to check this out: http://lug.ncsu.edu/

Impressive! We can probably use that as a base for our app. Let me ping
Sufyan about this.

I especially like how the newsfeed/blog part is integrated w/ the front
page.

I wouldn’t mind having a “multi-stack” architecture (where the front
page is powered by django, the events page by rails, etc).

Or are there existing modules (Rails gems, etc.) that you think we’d
want to use?

That’s the idea. If we were building this ourselves, we would probably
go with el-node

[aminb]

Thanks for that django/flask comparison, which is interesting. I’ve never used either of them but if it were me I’d use Flask based on my friend’s recommendation and so I could use his code and ask him things. YMMV. Why would we want a framework anyway though? What kind of code or application would we want to write (just event signups/calendar)? Or are there existing modules (Rails gems, etc.) that you think we’d want to use? If we’re writing our own stuff, would it be more in the Emacs spirit to use something written in Lisp, even if it’s not Emacs Lisp?

No worries Yeah Flask would be fine, but then we’d have to do a lot of stuff ourselves, which is already done in Django.

In terms of using a Lisp for website, I’d love to and I think Clojure has a nice ecosystem for web apps, however I’m not as well versed in Clojure, and I think it’d take more hassle.

Oh, as for a Lisp, I just remembered Hy, a lisp dialect embedded in Python. Maybe we could combine it with Django and write some parts of the website in Hy?

Oh that’s unfortunate about Gogs not having pull requests. I wonder how hard it would be to add them.

Apparently, it takes some work.

http://lug.ncsu.edu/ could be useful, though for something like Emacsconf I’m just not that excited by the idea of yet another Django/Flask/Rails site, and I’m always infuriated by site designs that eat vertical screen space with a fixed horizontal navigation bar. I don’t want to stand in anyone’s way about anything though. I do know Python pretty well so if this is just a “get things done” task, then that Django app is probably a good choice. If it’s also “be way cool” then I prefer things that are more hackerish (Rrix has me interested in Lnode even though it sounds terrible) and/or avant-garde (ur/web being an extreme example).

I wonder if there’s a framework for the nginx lua/redis modules, which apparently are blazing fast, especially with Luajit (Cloudflare uses it). And Lua can be fronted with Purescript (purescript.org) for that extra nerdery.

Usually, I’m all for hackerish crazy looking websites, as our own personal / project websites. But I think I’d prefer a more “normal” looking and casual site for a situation like this, a conference, and to be honest NCSU LUG’s website is quite a good candidate IMHO.

We could always work around that fixed nav bar / any other thing that we think about. My point is that the website already has a nice look / style to it and has implementations for stuff we need, we could use it as a base for building ours and save ourselves quite some time.

Amin, http://git.emacsconf2015.org/emacsconf/emacsconf2015 is just an org file slapped onto a git page expecting markdown, so it looks awful. The ahungry blog looks like this: http://ahungry.com/blog/I don’t know if there’s a way to get it to include the post texts on the front page. of course we could add that if it’s not there now.

I should’ve probably linked this instead of the source itself.

The blog looks cool (:

[phr]

Well I defer to you about the site appearance and so on. I like the current front page (plain text and static) and I figure we’re doing something wrong if we only want to do “normal” things and still have to write our own code ;).

Yes http://emacsconf.github.io/emacsconf2015/ looks like an org mode doc and I guess it’s what you get from org-publish or whatever it’s called. I personally like it and think it has the right feel for an emacs conference, but it’s up to you.

Thanks for the link about gogs pull requests. I wonder what’s going on with that. It seems ironic to me that gogs is hosted on github too. Come to think of it, is there a framework in Go that we can use? I’ve hacked in Go a little and it’s lower-level than Python but definitely has some attractive traits. It’s very easy to pick up, unlike Haskell.

http://offblast.org belongs to another guy I know and I like its look too, but I don’t think it’s quite the right look for emacsconf

[aminb]

Planning

Hey all, we’ve gathered a wiki page for planning, please take a look:

http://wiki.emacsconf2015.org/Planning

and tell us your opinion, if you agree with any of the proposed solutions or explain your own.

[samer]

Thanks for writing that up!

Do you know if there are any Django modules that we could use as the base of an events system?

[aminb]

Thanks for writing that up!

Do you know if there are any Django modules that we could use as the base of an events system?

No worries

There are a few “calendar” modules listed here. I haven’t tried any of them, but will do that soon.

[phr]

Based on irc discussion it sounds like we’re going to:

• replace front page rails app with lug django app
• put blog on front page
• install a django calendar/event module into the lug app, and also use it for stuff like emacs.sf (so it should be able to send out notifications of events)
• replace gitit with lug app wiki
• configure lug to use our existing postgres server that discourse uses
• (maybe?) reconfigure gogs to use postgres instead of mysql, it sounds like we’re currently running two different databases (plus redis)
• it should be possible to do ticketing/registration with the lug app, but deciding exact mechanism is temporarily postponed. (phr requested using some alternative instead of eventbrite).
• We don’t need postgresql replication but it would be nice to have some kind of periodic backups made automatically.
• rrix wants to put all server config stuff into ansible roles

[phr]

I notice the emacsconf server has five ruby processes totalling 1.2GB or so of resident memory. Is that some kind of apache-like preforking thing to handle concurrent requests in parallel? With our low traffic level, maybe it’s possible to configure discourse to use just 1 or 2 processes and free up a lot of ram. That would allow running in a 1gb container again, and some of those are available ridiculously cheap if you don’t mind crappy performance. It might be reasonable to migrate to something like that after the conference.

[test]

Ah, yeah, you can configure the number of processes that discourse uses.
Right now we’re using the “good” default of 5, but we should try lowering
it.

Thanks for pointing this out!