#+STARTUP: showall hidestars
#+TITLE: Example of making and managing a website with emacs org-mode
#+LINK_UP: ../index.html
#+LINK_HOME: ../index.html
#+STYLE: <link rel="stylesheet" type="text/css" href="../css/stylesheet.css" />
#+OPTIONS: toc:nil

* Comments.
[[http://danamlund.wordpress.com/2009/12/23/danamlund-dk-org-mode-source/][Comments]].

* News
  - *20 Mar 2010*: Published.

* Description
I explain how I make and manage this website using /org-mode/ and a bit
of other /emacs/ features. I talk about the problems I needed to solve
in order to use /org-mode/ to host this specific website. This means
setting /org-mode/ up to convert a set of files to html, how to write
documents in /org-mode/, and how to automatically update the website
using some /elisp shell scripting/ and /lftp/ to mirror the ftp server.

* Problems
  1. Setting /org-mode/ up to convert to html (publishing).
  2. Writing documents using /org-mode/.
     1. Code highlighting used weird colors because of emacs
        color-theme different from website colors.
  3. Automatically update and upload the website.
     1. Generate a directory listing html file (sitemap).
     2. Don't upload /svn/ '.svn' directories.

* org-mode publishing setup
I followed [[http://orgmode.org/worg/org-tutorials/org-publish-html-tutorial.php][Worg's org-mode publishing tutorial]] when setting up my
website. My text here will be less complete as I will only talk about
the features I used.

In =~/data/danmalund.dk/org= I have my source files (meaning org, css
and any other files that should be processed or simply copied to the
website directory).

In =~/data/danamludn.dk= I have my published files, meaning the
html-files that make up the website.

So to set this up in org-mode I have the elisp code:

#+BEGIN_SRC elisp
  (org-publish-projects
       '(
         ("danamlund.dk-notes"
          :base-directory "~/data/danamlund.dk/org"
          :base-extension "org"
          :publishing-directory "~/data/danamlund.dk"
          :recursive t
          :publishing-function org-publish-org-to-html
          :headline-levels 999
          :auto-preamble t
          
          :headline-level 999
          :section-numbers nil
          :sub-superscript nil
          :style "<link rel=\"stylesheet\" type=\"text/css\" href=\"./css/stylesheet.css\" />"
          :author "Dan Amlund"
          :email "danamlund@gmail.com"
          )
         ("danamlund.dk-static"
          :base-directory "~/data/danamlund.dk/org"
          :base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
          :publishing-directory "~/data/danamlund.dk"
          :recursive t
          :publishing-function org-publish-attachment
          )))
#+END_SRC

There are 2 "projects", one that converts org-files to html-files. And
one that just copies every file over. I will explain some of these.
  * =base-directory= is where the source-files (org-files in this
    case) are located.
  * =publishing-directory= is where to copy the processed input
    files.
  * =publishing-function= is how to process the input files (convert
    /org/ to /html/ in the first case, just plain copy in the second).
  * =headline-levels= depth (amount of =*='s) to threat as new
    sections. (don't see why anyone would need less than infinite)

The rest (=headline-level=, =section-numbers=, =sub_superscript=,
=style=, =author=, =email=) are default values for settings related to
org files, they can be overwritten by chaning the value in each
orgfiles.

Note that this a call to the org-mode function =org-publish-projects=
which publishes the input projects. So executing this function (move
cursor after last =)= and C-x C-e) will create the website files.

Note that my org-files are located in a sub-directory of where the
published website is located. This doesn't cause any errors as long as
I don't overwrite any files by having a /org/ directory inside my
source directory.

* org-mode documents
[[http://orgmode.org/worg/org-tutorials/org-publish-html-tutorial.php][Worg's org-mode publishing tutorial]] also talks about how to organize
org files in a published project. I will discuss how I set it up for
this website.

As an example I will use the org files describing this html file which
is called
#+HTML: <a href="../org/emacs/orgsite.org">orgsite.org<a>
.

At the very beginning of orgsite.org we have
#+BEGIN_SRC org
#+STARTUP: showall hidestars
#+TITLE: Example of making and managing a website with emacs org-mode
#+LINK_UP: index.html
#+LINK_HOME: index.html
#+STYLE: <link rel="stylesheet" type="text/css" href="./css/stylesheet.css" />
#+OPTIONS: toc:nil
#+END_SRC org

These are options to org-mode specifying meta-data like the title, how
the file is shown in emacs like the startup, and how the published
page should look like the options. The =title= defines the title of
the document and is used to set the title of the html document. The
=startup= defines the =showall= and =hidestars= options, which shows
all text and hides all but the last star (in section definitions). The
=options= sets the value of =toc= to true, meaning a table of contents
will be created when converting the file to html.

Some meta-data (like author and email) are left out as default values
for those are defined in the org-mode publishing project.

We define a stylesheet link in one of these options. The url to this
file has the match any possible sub-directories of the org-file and
it's published html-file. As orgsite.org is inside one sub-directory I
have added =../= in front of the url. I stole the stylesheet from
http://orgmode.org/worg and then removed alot of the fancy stuff I
didn't really understand, I just needed the UP and HOME links and the
floating table of contents.

The actual page setup of orgsite.org consist of sections and
subsections which are defined by 1 or more =*= followed by a
title. Links are defined inside a bunch of square brackets. These
brackets are hidden when viewing the file in emacs and can be edited
by moving the cursor over the link and C-c C-l. Fonts are defined by
surrounding text by various symbols (* for bold, / for italic, = for
fixed-width). 

I define code highlighting with the same syntax as in the beginning of
the file (=#+BEGIN_SRC elisp= and =#+END_SRC=). The =elisp= means that
the highlighting colors will be fetched from =elisp-mode=, later i use
=org= which highlights according to =org-mode=.

In the end of orgsite.org I have an empty section which closes any
sub-sections before printing the last part of the html-file which is
the author and timestamp. I also add a horizontal line to seperate the
page from the author and timestamp part of the html. Note that I use
=#+HTML:=, if I didn't then the < and > signs would be converted to
printable characters and not be parsed by the browser.

** Fixing code highlighting when using colors different from the website.
My emacs have a black background and white text. If I convert
org-files to html code-highlighting will probably look weird, meaning
having black background color for some keywords and generally using
colors with a low contrast to the white background of th website.

To fix this I make some elisp that automatically changes the colors to
white background and black text, then convert org-files to html and
then change the colors back.

#+BEGIN_SRC elisp
  (progn
    (set-background-color "white")
    (set-foreground-color "black")
    (org-publish-projects
     '(
       ("danamlund.dk-notes"
        :base-directory "~/data/danamlund.dk/org"
        :base-extension "org"
        :publishing-directory "~/data/danamlund.dk"
        :recursive t
        :publishing-function org-publish-org-to-html
        :headline-levels 4
        :auto-preamble t
        
        :headline-level 4
        :section-numbers nil
        :sub-superscript nil
        :style "<link rel=\"stylesheet\" type=\"text/css\" href=\"./css/stylesheet.css\" />"
        :author "Dan Amlund"
        :email "danamlund@gmail.com"
        )
       ("danamlund.dk-static"
        :base-directory "~/data/danamlund.dk/org"
        :base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
        :publishing-directory "~/data/danamlund.dk"
        :recursive t
        :publishing-function org-publish-attachment
        )))
    (set-background-color "black")
    (set-foreground-color "white"))
#+END_SRC

* Automatically update and upload
To automatically update and upload my website I convert org-files to
html, create a directory index, remove .svn files and then use lftp to
upload any changed files to the website hosting server.

I will describe what the elisp part of
#+HTML: <a href="../org/index.org">index.org<a>
.

** Easily converting org-files to html
I use the same elisp I showed earlier to do this. I keep it in my
index.org file under the section named /COMMENT/ which isn't included
in the html output. I then simply execute the elisp (C-c C-e) when I
need to generate the html files.

** Generating directory listing html file
For some reason my hosting site doesn't allow directory listings
(disabled by default and not changeable through .htaccess). I fix this
by writing a simple elisp script that makes a sitemap with links to
all the files in a given directory.

#+BEGIN_SRC elisp
  (progn
    (defun my-ls-html (dir &optional prefix)
      "Return a string of html showing recursive directory listings."
      (let ((out ""))
        (dolist (f (directory-files dir))
          (if (not (or (string= f ".") (string= f "..")
                       (string= "." (substring f 0 1))))
              (let ((ff (concat prefix f)))
                (if (file-directory-p (concat dir "/" f))
                    (setq out (format "%s<li>%s:%s</li>"
                                      out f (my-ls-html 
                                             (concat dir "/" f) 
                                             (concat prefix f "/"))))
                  (setq out (format "%s<li><a href=\"%s\">%s</a></li>"
                                    out ff f))))))
        (concat "<ul>" out "</ul>")))

    (with-temp-file "~/data/danamlund.dk/org/index.html"
      (insert (my-ls-html "~/data/danamlund.dk/org"))))
#+END_SRC

It's not pretty code but it does a simple enough job that it doesn't
need to be.

I first define the function which outputs a string, then I write that
string to a file using the builtin =with-temp-file= function.

** Setting up lftp
/lftp/ is just a simple ftp client I found that can mirror upload
sites. This feature is simple to decrease the time it takes to upload
a change in the website.

/lftp/ itself can be passed, as a parameter, a series of commands to
connect and start mirrroring the website. To execute this automatically
using emacs I use the =shell= and =comint-send-input= commands.

#+BEGIN_SRC elisp
  (progn
    (shell "lftp")
    (insert (concat "lftp -c 'open danamlund.dk@danamlund.dk;"
                    "mirror -R -e ~/data/danamlund.dk_tmp /'"))
    (comint-send-input))
#+END_SRC

This code asks for the password when executed. I could have included
the password as a part of the lftp parameter but then I wouldn't be
able to safely share my index.orgfile.

** Don't upload .svn directories
Ftp is really slow the more files and directories you have. Mirroring
saves some time, the client still need to look through all
directories. To speed up ftp mirroring I decided to remove .svn files.
Unfortunately /lftp/ doesn't seem to have a feature to ignore certain
files, so I fixed this with some shell commands instead. I basically
just copied the entire generated website to a temporary location and
then removed /.svn/ files using /find/ and /xargs/.

#+BEGIN_SRC elisp
  (progn
    (shell-command "rm -Rf ~/data/danamlund.dk_tmp")
    (shell-command "cp -RLf ~/data/danamlund.dk ~/data/danamlund.dk_tmp")
    (shell-command "find ~/data/danamlund.dk_tmp -name '.svn' | xargs rm -Rf"))
#+END_SRC

** Putting it all together
As can be seen in index.org I have combined these features into two
operations, one to generate the website and one to upload it. It is
usually a good idea to check the website before uploading it to the
public.

#+BEGIN_SRC elisp
  (progn 
    (require 'org-publish)
    (set-background-color "white")
    (set-foreground-color "black")
    (org-publish-projects
     '(
       ("danamlund.dk-notes"
        :base-directory "~/data/danamlund.dk/org"
        :base-extension "org"
        :publishing-directory "~/data/danamlund.dk"
        :recursive t
        :publishing-function org-publish-org-to-html
        :headline-levels 4
        :auto-preamble t
        
        :headline-level 4
        :section-numbers nil
        :sub-superscript nil
        :style "<link rel=\"stylesheet\" type=\"text/css\" href=\"./css/stylesheet.css\" />"
        :author "Dan Amlund"
        :email "danamlund@gmail.com"
        )
       ("danamlund.dk-static"
        :base-directory "~/data/danamlund.dk/org"
        :base-extension "css\\|js\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf\\|deb"
        :publishing-directory "~/data/danamlund.dk"
        :recursive t
        :publishing-function org-publish-attachment
        )))
    (defun my-ls-html (dir &optional prefix)
      "Return a string of html showing recursive directory listings."
      (let ((out ""))
        (dolist (f (directory-files dir))
          (if (not (or (string= f ".") (string= f "..")
                       (string= "." (substring f 0 1))))
              (let ((ff (concat prefix f)))
                (if (file-directory-p (concat dir "/" f))
                    (setq out (format "%s<li>%s:%s</li>"
                                      out f (my-ls-html 
                                             (concat dir "/" f) 
                                             (concat prefix f "/"))))
                  (setq out (format "%s<li><a href=\"%s\">%s</a></li>"
                                    out ff f))))))
        (concat "<ul>" out "</ul>")))
    (with-temp-file "~/data/danamlund.dk/org/index.html"
      (insert (my-ls-html "~/data/danamlund.dk/org")))
    (set-background-color "black")
    (set-foreground-color "white"))
  
  (progn
    (shell-command "rm -Rf ~/data/danamlund.dk_tmp")
    (shell-command "cp -RLf ~/data/danamlund.dk ~/data/danamlund.dk_tmp")
    (shell-command "find ~/data/danamlund.dk_tmp -name '.svn' | xargs rm -Rf")
    (shell "lftp")
    (insert (concat "lftp -c 'open danamlund.dk@danamlund.dk;"
                    "mirror -R -e ~/data/danamlund.dk_tmp /'"))
    (comint-send-input))
#+END_SRC

#+HTML: <hr />