Create your own map with OpenStreetMap data on Ubuntu - Phase 2: Importing and introduction to Carto

In my previous blog post, we set up a postgres database with OSM data and we installed TileMill, now it's time to play with it's settings. Remember, to start TileMill, you should execute the tilemill.js script from the directory where you installed TileMill.

TileMill is very GUI centered, so I hope I will include enough screenshots.

Importing

First, we create a new project, let's call it Test.

If everything is right, you see a grey world and a white see.

Now we have to add the layers. I hope you know how OSM data is tagged, if you don't, please look at the features list in the wiki.

Let's try to make a basic map with streets, labels and some POI.

We will begin by adding all data in layers like it is in tables now.

Click on the '+' symbol to add a layer and fill in the details like below:

The database tables that contain all data are called

planet_osm_roads
planet_osm_line
planet_osm_polygon
planet_osm_point

I don't personally like a white see, so let's make it blue.

Map {
    background-color: #aaf;
}

#world {
    polygon-fill: #eee;
    line-color: #ccc;
    line-width: 0.5;
}

As you see, the Carto language they created is css-like. But it has advanced features like nesting.

Ways

Let's draw the motorways.

#line [highway='motorway'] {
    line-color: blue;
    line-width:4;
}

Here you see the selection in action. You don't draw all lines, only those with a tag highway=motorway. But you can also nest selections. Say e.g. that we want a wider way if we zoom in.

#line [highway='motorway'] {
  line-color: blue;
  line-width:4;
  [zoom>15] {
    line-width:10;
  }
}

In that example, you see that previous commands are overwritten. You can also select on a tag being not empty. We can e.g. draw all ways with the following code:

#line [highway!=''] {
  line-color:grey;
}

Polygons and points

Drawing polygons works in a comparable way:

#polygon [landuse='residential'] {
  polygon-fill: #dedede;
  line-color:brown;
}

And for points, you can draw a symbol.

#point [amenity='pub']{
  point-file:url('http://svn.openstreetmap.org/applications/rendering/mapnik/symbols/pint.png');
}

Off coarse, it is best to set up your own local set of map icons.

Labels

Most maps also have labels (streetnames, village names ...). For the names, you have to watch out to always define the font, otherwise it won't be rendered.

#point [place!=''] {
  text-face-name:"Andale Mono Regular" ;
  text-name:'[name]';
}

 In the above example, we render the tag 'name', it is also possible to render another tag, like for house numbers, you should render the tag 'addr:housenumber'.
Predefinitions

If you have to use the same value (like a font or a color) over and over again, it might be good to define "global variables". If you change your mind, that's easy to adapt.

You can create a global variable at the beginning of the mss file like that:

@residential: #dedede;

And you can use it easily as follows:

#polygon [landuse='residential'] {
  polygon-fill: @residential;
  line-color:brown;
}

Reference

All fonts and used colors can be found in a separate part of the screen.

If you need extra commands (like rendering a halo around the text for better visibility), you can also check the Carto reference.

Next post

The osm2pgsql script we used in part 1 is a lossy script. That means it only imports certain tags to the database. As an example, it doesn't load any rcn_ref tags or bicycle routes. The next time, we're going to see how we can get around that.

We will also take a look at SQL selecting instead of selecting with the Carto language.

Create your own map with OpenStreetMap data on Ubuntu - Phase 1: setup

In this tutorial, we're going to use TileMill, an easy frontend to Mapnik, on Ubuntu. So first you have to install TileMill and Mapnik. There are good and up to date commands given on the TileMill site: http://tilemill.com/

Now you need to get your data ready. First of all, you need to download the data. If you only need data for one small city, you can use the export tab on the OpenStreetMap website. Choose to export to OpenStreetMap XML data.

If you need more than one city, the api is probably not going to accept such a big request, but than you can use the cloudmade OSM downloads. Here again, choose to download osm files, but these here are zipped. So you need to unpack them before you can use them. If you need the whole world (a whopping 220 GB of data, compressed to 16 GB for downloading), you can check out different ways to get the planet file: http://wiki.openstreetmap.org/wiki/Planet.osm. Reading the entire planet takes, even on strong servers, serveral days.

Now we have to set up a database and put our data in it. Postgres has already been installed if you followed the installation tutorial of the TileMill page, but we still need to configure it.

The instructions come from the OSM wiki and are confirmed to work for Ubuntu 11.04.

(note that long lines are split in two in blogger. if you have problems with this, use the OSM wiki instead.)

sudo -u postgres -i -H
createuser -SdR gisuser
createdb -E UTF8 -O gisuser gis
createlang plpgsql gis
psql -d gis -f /usr/share/postgresql/8.4/contrib/_int.sql
psql -d gis -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
# to populate the table spatial_ref_sys (mandatory for use with osm2pgsql):
psql -d gis -f /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql
psql gis -c "ALTER TABLE geometry_columns OWNER TO gisuser"
psql gis -c "ALTER TABLE spatial_ref_sys OWNER TO gisuser"
psql gis -c "ALTER TABLE geography_columns OWNER TO gisuser"
exit

Now you need to edit a configuration file before that user can login to the database.

sudo gedit /etc/postgresql/8.4/main/pg_hba.conf

Change in the last lines that should look like

local all all ident
host all all 127.0.0.1/32 md5

the words ident and md5 to trust.

Now you need to reload the database and login to test it.

sudo /etc/init.d/postgresql reload 

psql gis gisuser

With the command \d, you can see the databases that are created and with \q you can exit.

Now you need to get the fresh OSM file into a pgsql database. This happens with the osm2pgsql program.

sudo apt-get install osm2pgsql

And finally put the data in the database

osm2pgsql -m -d gis -U gisuser  data.osm

 Where dat.osm is off coarse your file you downloaded. You don't need to unpack it. If it's a big file, this process can take a huge amount of RAM and a lot of time.

Now that you have imported the data, you can add it to a layer on your TileMill. If your TileMill server isn't running anymore, start it with the tilemill.js script in the directory where you installed TileMill.

Now go to http://localhost:8889 and add a new project. Open the project by clicking on it, you will see a world map with all boundaries and sees drawn. It will also open a stylesheet file for that worldmap.

Add a layer, choose for Postgre SQL and fill in the details.

The id is the name of the layer. The database name is gis (unless you changed it) and the user is gisuser. In the query, you put the table created by osm2pgsql. These are the following tables:

planet_osm_point
planet_osm_roads
planet_osm_polygon
planet_osm_line

So you cannot enter the lines and polygons in one layer. You need to add 4 different layers if you want all features.

I've called the layers point, roads, polygon and line. The names suggested by osm2pgsql.


Now your data is in the TileMill application. The next post will be easier and I will try to create some maps with color and icons.

Navit for Android vs OsmAnd

In this post I will try to compare Navit for Android and OsmAnd. But before I can compare them, I need to explain what they are.

Goal of the two projects

The Navit project wanted to give an open source alternative for various proprietary navigation systems. It was founded in the pre-smartphone area, so navigation systems were separated from other funtions or it was done in a car-computer. Because the team wanted to support as much as possible different devices, it is witten in C and it uses (almost) no libraries. It was also before OSM was well known and thus it also supports different maps. As you see, the system is very modular. The Android version of it just a very thin layer of Java GUI on top of the C code.

The OsmAnd project wants to give free (as in beer and speach) navigation to everyone. But it can only be used on an Android phone, so you should replace "anyone" with "anyone that owns an Android". That immediately makes it less free as in beer but still completely free as in speach. To achieve that goal, they (almost) exclusively use OSM data and try to minimize the mobile data costs.

Various features

As Navit is made for all kinds of devices, it makes no use of any kind of connection (except for downloading the map files the first time you use it). So it uses no tiles that come from somewhere else, it uses no external routing engine, no traffic info. Its just you and Navit that reads your map. As an effect of this "no external data" policy, a lot of work has been put into the different engines. Both, the rendering and the routing engine are completely configurable via the XML configuration file. different color styles and weights can be given to different types of road. Even the menu (the native menu, not the Android part of the menu) is configurable via the XML. The widgets you put on the default screen (compass, speed camera notifier, distance and time to your target, 2D ...) can also be configured in the XML file.

OsmAnd doesn't come with such a customisability, but it offers more options through it's menu. You can choose YOURS or Cloudmade routing engine (an internal routing engine is being developed right now). You can choose different tile suppliers (the well known big suppliers as Microsoft and Google, but naturally also smaller OSM tile suppliers) or use the vector data engine with about five themes to choose from.

Both apps offer a build-in search engine. With Navit you have search by town and street and for the Android version you also have fuzzy search (you type a clue and Navit searches it, a bit like Google search or Nominatim), the fuzzy search is off coarse not so elaborated like Google or Nominatim (a smarphone does not have as much memory like a search server) but it workes good in regions where OpenStreetMap has good maps. OsmAnd also offers internal search for town, street and building (though not all kinds of tagging are recognised, so you might be in a region where a lot of buildings are drawn, but none of them is found) alternatively for the building search, you can also search for an intersection with another street. Next to the internal search, OsmAnd also offers a fuzzy search using the Nominatim api.  

Data usage

Say now, as example that you want to go somewhere, be guided by a voice and use as less data as possible.

Navit does everything internally, so uses no data and you can use tts for the voice guidance. You do need to use an external tts engine for the speaking. There is eSpeak for Android and easyTts availiable in the market, they are both derived from the eSpeak project. EasyTts has more recent data, but not as much languages.

In OsmAnd, when you uses as less data as possible (pre-fetched maps, vector rendering ...) then you still need to use an online routing engine. One route typically costs less than 30kb. That's pretty cheap, but if you don't have a connection, it won't help you. If you change your route while you're driving, you need to download a new one. For all the other parts (voice commands, rendering the route ...) OsmAnd can do in offline. Spoken commands can be delivered by using CHK voice files (like used in tomtom) or by downloading voice files from within the program. 

Next to the online routing engine, OsmAnd can also use pre-created GPX files. So you can traceback some older route or follow a route that you created online. This costs no data. 

So to resume, you can use OsmAnd without data connection, but then you need to be prepaired (create a GPX file before you start or create your route while you're on wifi), otherwise you will use a very little bit of data. With Navit, you can be always offline.

Performance

Let's now come to the performance. I only have a basic Android phone, so it's likely that a reader will see better performance than I do. First of all, the Navit only feature: route calculation. 

Route calulation in Navit work pretty well. On my phone I can calculate routes that are up to 800 km long (so that uses a lot of memory). Off coarse it is not as long as routes created by online routing engines, but I believe 800 km is good enough for all usage.

If I compare the vector rendering of both OsmAnd and Navit, I see that they are both good but not perfect. When you drag a map, it takes a while before it gets rendered on both apps. By leaving out a features in the Navit XML file, you can get a lot better performance, but off coarse you lose functionality (as you see less on your map). 

The structural search in Navit is faster than the structural search in OsmAnd, but the fuzzy search in Navit is slower. This doesn't make such a difference, since you don't search that much and the typing still takes longer than the searching.

Installation size

In this category, OsmAnd is clearely the winner. The Navit app is very big and the maps are also big. This is only a one-time-download, but it takes a lot of space on your sd card.

Interface

"De gustibus et coloribus non disputandum est"

OsmAnd is native Android, so it follows the standard way you're used to on your Android phone. Navit isn't and thus it has a bit different UI and looks less good compared to your other apps. Navit has been Androidified a bit, so you have an Android menu with the most basic things (like search, navigate to ...) but the more difficult settings still need to be done in the original menu. This makes it a bit weird to work with but once you're used to it, it works fast.

In OsmAnd you can change a lot of settings via the menu, so you really need a good structure in the menus. This is unfortunately not completely so. Some settings are a bit in an illogical place. I don't think I could do it any better, but this is still a problem. The most used features are where you expect them tho.

Conclusion

Both apps are really good and it depends on your use, if you want to be completely offline, you could try Navit. If you want a low data usage but you're still able to use data, OsmAnd is the best one.

My Android and pure AOSP

As I talked about in my post "what is android without google", it is possible to get a pretty functional phone with only OSS apps.

This was only in theory and I wanted to try it with my own phone. I already ran CyanogenMod 7 on my phone. This is quite free from proprietary vendors (you don't get the proprietary sense framework for example). But still, the google marked was shipped with it and some other proprietary apps.

So I switched to the pure AOSP compiled by Fydor.cn on XDA. The only proprietary things left are some drivers. But I don't think this will bother anyone.

So I putted the pure AOSP on it and installed the FDroid apk. I decided to only work with apps that are in the FDroid repository. This is even more limited than saying "only OSS". The phone and messaging app are part of the AOSP, so I didn't have to look for that.

First of all, I wanted to get my mail to work. I saw that there was a mail app, but as far as I see, it only works with exchange, not with pop or imap. So I took the K-9 Mail app from FDroid, gave it my gmail address and password and it connected without a problem.

Secondly, I also wanted to be able to chat via a jabber account. So I installed beem. There I got a bit of a problem, for now, it seems that it's only one-way communication. Maybe that's because I'm also tethering with my phone, but I don't seem to be able to send from my computer to my phone. I'll try it again when I have decent internet.

A calendar app was also no problem. with aCal, I could connect to my google calendar (although I had to check the wiki to see how to do it). There is a calendar app in the AOSP project, but it seems only able to connect to exchange calendars.

As you can expect, there is no immediate way to sync contacts (although exchange syncing is supported again), but I could import a vCard in the contacts app.

For the rest, I installed a pretty good pdf reader, a text editor and an RSS reader. With that rss reader, I was able to read google news (the news and weather app seems not available too).

I don't think you need anything more, I have camera working, FM radio, a music player, a file manager, a browser and navigation software. There are also some games available in the FDroid repo (like tetris or sudoku), but you don't really need them.

Maybe there is one thing missing: a screen calibration app. It is difficult to use my keyboard because the keys at the border of the screen don't work very good. There are some calibration apps available, but it's difficult to see if they are open source or not. I think I could configure it manually (since I have root access) but I don't really want to get into that.

So to answer my question from the previous time, pure open source Android without Google is a bit more difficult to set up (you need to set up each app separatly) but once configured, it is just as functional like a normal Android phone. Cyanogen does do a good job to give you as much open source as possible without giving up the easy configuration.

HTC making their sense framework open source

HTC is making their sense framework open source. Next to that, they also release a "HTC Opensense SDK". So now you can build apps based on the HTC sense framework.

This has two results.

First of all will all phone makers and invidual users be able to use the Sense interface for their phone, second will all programmers be able to use the framework as extra libraries. The disadvantage is that apps based on the Sense framework will be heavy for those who are not using Sense as their interface. You could compare it with loading up a KDE app when you are on Gnome. The loading of the extra KDE libraries asks a lot of time and a lot of memory. Open sourcing their framework is a good thing, but I don't think they should have made it in the first place. If they wanted extra libraries, they should have written some patches and sent it to main Android developers. What they are doing now is basically forking Android.

This is just my two Sense about the topic.

What is Android without Google?

Android is an Open-Source OS ( the Linux kernel with an apache-licensed GUI and libraries on top of it). But when you buy an Android phone, you almost always get a bunch of closed-source google apps with it. The problem with this is that google has complete control over its apps. Google could for instance say that you may not install the market on a custom rom anymore. Google can also exclude some hardware vendors from using its apps (like it seems to do with the Arnova tablets).

So I would like to see if there are OSS replacements for those apps. Lets start with the most problematic app: the market. As you all probably know, one of the main strengths of a lot of Linux distributions is there package management. Ubuntu would be less practical to use if you couldn't just apt-get install a package. The same is true for smartphone OS. their quality is expressed in the number of apps that are availiable in the market. The problem is that the Android market is a proprietary Google app and thus no part of Android.

 There is a good list for all appstores (for all OS) on WIP connector. Lets give an overview of a few Android appstores.

• Amazon app store: not open source, you have to pay $99 per year to get your app in the store and if you want to post a paid app, then you get 70% of the price. For customers, the Amazon app store does offer a malware check before apps are included. Amazon also has a large number of apps and other stuff.
• Aptoide client is an OSS application market. You can add your own repositories (and even create your own ones), but the included repo has currently over 1400 apps. Aptoide also has support for user ratings, although currently most apps are unrated. Aptoide does not support paid apps.
• Fdroid is an OSS alternative but only gives access to OSS apps. This has as positive side that OSS apps are promoted. If you want more apps, the same repositories availiable for Aptoide can be used in Fdroid. But it does not support user ratings yet (maybe in the future). If you want your application in Fdroid, you can simply fill in their submit form.
• SlideMe: closed source, no subscription fee and you get about 90% of the price of your app.

As you see, there are replacements (even OSS ones), but not with as much applications as the original Android market.

Now that the most important app is discussed, lets talk about the other ones. All apps can be found on google.

• The search box and Goggles: Google and search match quite well, especially if it is on an OS that they developed. But as alternatives, you could use DuckDuckGo for websearch and Quickdroid for internal search (apps, contacts ...)
• Gmail: The mail app in the Google apps is only limited to Gmail addresses. But there is a better app for this: K9 mail, it's open source and accepts imap, pop3 and exchange mail. The only disadvantage is that it doesn't work with the priority inbox Google created.
• Google maps with latitude and navigation: I work a lot on OpenStreetMap, so for this one, I know there are a lot of alternatives. At least for the maps with navigation. Geeks don't seem social enough to use Latitude :P. The most complete alternative is Navit. It's an Open-Source offline navigation application. There are also other open source apps or free proprietary apps that do online routing using OSM data. The list can be found on the OSM wiki.
• Youtube: If you have a device with flash, you could off coarse use the browser to view youtube movies. But since youtube is a google service, you probably won't find any better app. But there are alternative apps for other video sites like dailymotion, but not a lot of OSS apps.
• Buzz: Buzz doesn't seem really popular, so I don't expect other apps for it.
• Google voice: As far as I know (I'm not in the USA, so I can't test google voice), it is just a VOIP app with some integration with your google account. There are other VOIP apps, and I know Jitsi is working on an OSS VOIP app. There are also OSS SIP clients availiable.
• Google talk: Google talk uses the jabber protocol, so there are other apps offering this and if the Jitsi app comes out, it will certainly offer calling over the jabber protocol. For now, there is Beem you could use.
• Google calendar: The Google calendar app is open source, but the underlying sign in isn't. So there is no possibility to use the calendar app  without the google apps. There is however the app AnCal. This one is open source but doesn't offer syncin. The app aCal is another calendar but this one supports syncing to a CalDav server.
• Google contacts: the contacts app is open source and usable without signing in to Google (but then it doesn't offer syncing). A fork of it is also present in the Cyanogen mods.
• Google Finance: I suppose there are alternatives, but I don't use any.
• Google shopper: I don't think there are alternatives for this.
• Blogger: A lot of alternatives for this, not only limited to the Google service of Blogger but also for other services. There are however no OSS alternatives for Blogger, but you can still use the browser interface.
• Translate: I don't use it, but I guess there are proprietary alternatives.

So as a summary, I could say that most apps have an OSS concurrent (exept for the pure google services) and for the Market you even have a choice to have one with only OSS apps in it or one with all kinds of apps.

So if anyone is considering to use Android without Google, or if the gapps don't work on your phone, you see that there are alternatives.

LaTeX and a HTML5 book

Maybe you've seen the HTML5 book google made about "the internet": http://www.20thingsilearned.com/.

Well, they open-sourced the code two days ago, it's at google code. But I'm wondering how difficult it would be to build a LaTeX2html5 script which exports (just as latex2html) a LaTeX file to html, but this time using their code, so you get a real good looking book. It should not be too difficult, but once I have the time, I certainly will try to make that.

Off coarse, the images won't be part of the book and the only real problem I see is on how to do the splitting in pages. For the plain LaTeX text, I could use the latex2html script, but then I need to calculate the height and break the page at certain heights. This will be difficult to do when commands like \input are used or when there are a lot of formulas and figures. But nothing is impossible and it seems a fun project.