Deploying OpenStreetMaps


Hey everyone,

OpenStreetMap is the map of the world with open license and freedom. I deployed OpenStreetMaps on my system after seeing a lot of people having a lot of troubles deploying them. I used Ubuntu 16.04, i3 (2-cores, 4-threads), 4GB RAM system for this and this link as source: https://www.linuxbabe.com/linux-server/openstreetmap-tile-server-ubuntu-16-04.

You need to download map file for your region of interest. I download India’s map (~400MB pbf file). You can download from here and of course can choose another region. Now let’s continue as the tutorial source I mentioned takes us. I am not going to repeat all of that (DRY), it’s easy. After installing some dependencies and postgresql, I created osm user, created db, created extensions.

Download Map Stylesheet from https://github.com/gravitystorm/openstreetmap-carto/releases and download the latest version which at this time is v4.0.0.

Next, you are going to import the download map file into the PostgreSQL database. If you have a small amount of RAM, then it’s recommended to create a swap space to speed up the import process. I created 2GB for me with the fallocate command. I noticed that fallocate doesn’t support NTFS file system (and a few other too). I tried creating on ext4 file system on another drive and it works fine. Then we mount it as swap memory. You can check it using: cat /proc/swaps or top commands.

Now after installing the osm2pgsql tool, the actual importing process starts. Following command does it all:

osm2pgsql --slim -d gis -C 3600 --hstore -S openstreetmap-carto-2.41.0/openstreetmap-carto.style great-britain-latest.osm.pbf

3600 (3.6GB) here is the amount memory you want to allow the import process to be taken. You can try taking a little more. I tried the double of it and it gave error. -d is for database name (gis). Change the files and folders according to your own system e.g. openstreetmap-carto-2.41.0 will probably be different on your system and great-britain-latest.osm.pbf will be name/path to your map file you downloaded. You can also add additional argument –number-processes=4. I added because of the 4 threads of my CPU. Hence the above command in my case becomes:

osm2pgsql --slim -d gis -C 3600 --number-processes=4 --hstore -S openstreetmap-carto-4.0.0/openstreetmap-carto.style india-latest.osm.pbf

It takes time to import. So, go out and enjoy life (plug in the charger/if on server, use tmux). Okay, it’s finished finally.

Now time to compile the mod_tile module required to serve the tiles. If you run into some error during the compilation, refer to this for installing some missing dependencies. After it’s done, let’s move forward.

Generate Mapnik Stylesheet

You’ll be asked to go to openstreetmap-carto-xxx directory and execute get-shapefiles script. In the newer version, they have converted the bash script to Python script and moved it to script directory. That’s why it’s recommended to follow the official guide. So go to script directory and execute it as:

python get-shapefiles.py

And then run in previous directory,

carto project.mml > style.xml

Above command (carto) gave me error:

carto: Unexpected token s

It was due to very old version of carto installed. Found the hint at https://github.com/gravitystorm/openstreetmap-carto/issues/2583 to use higher version (>0.16) of carto.

Installed latest version:

sudo npm install -g carto

You’ll need to install nodejs for this (npm – node package manager) to work. Then I executed carto again and it worked, despite giving some warnings like:

Warning: using the name attribute for layers (like building-text here) is deprecated and will be removed in 1.0.0. Use id instead.

But it’s OK. So if yours one is still giving the same error (carto), try logging out and log in again. Check if your carto version is updated. Mine shows:

$ carto -v
 carto 0.18.0 (Carto map stylesheet compiler)

Move on to configuring the renderd and edit /usr/local/etc/renderd.conf and edit your details carefully as per your own system.

After configuring Apache, if you point yourself to the browser at http://localhost/osm_tiles/0/0/0.png (or whatever ip) and get a image showing a world map, then you are lucky. If you did’t, you probably have to check Apache logs (/var/log/apache2/error.log) and hence this makes you more lucky. In my case, the tail of log file contains:

/var/run/renderd/renderd.sock No such file or directory

So, I checked and there wasn’t renderd directory. So I created on and given appropriate permissions. Then I executed again and I got the map image. Congratulations!

Display Your Tiled Web Map

Now, it’s time to see the map zoom-able and span-able. There are two JavaScript libraries available: OpenLayer and Leaflet. You  just need to download and extract it to your root of your web server like (/var/www/html). Edit the index.html file and put the OpenLayer specific code. Make sure to edit lines like: http://your-ip/ol.css. Replace your-ip with your ip-address. If you are testing on your local system, then put localhost instead. Point your browser to localhost and you should see the map. Now the tiles are generated on-the-fly. If you want to pre-render the tiles so that your processor doesn’t do processing everytime you zoom-in then you can do it via:

render_list -m default -a -z 0 -Z 10

This command will take some time. And yes, you can stop and resume it anytime. It won’t start from start again. It will generate tiles for various zoom levels.

The issue I was having at this step was no map was being shown. Seeing the apache logs disclosed the error:

socket connect failed for: /var/renderd/renderd.sock with reason: Connection refused.

I tried starting the renderd:

sudo systemctl start renderd

If you have nginx installed and running, stop it and change ports to avoid conflict with apache. Then restart apache and renderd. It should solve it.

Thank you.

See “less” output at a time


There are a lot commands/tools that produce a lot of output that we can’t see at once (and without scrolling). Most of the commands on terminal can be made to show us whole output page by page using “less” command.

less command_here

less ls -R ~/

or you may pipe the output to less command,

ls -R ~/ | less

 

or similarly, to see the contents of a file:

cat file_name | less

There are many commands/tools that have their own prompt.

 

For IPython, you may do:

%page variable_name

 

Similarly for MySQL,

pager less -SFX

Or you may also try:

SELECT * FROM sometable \G

 

Above commands will show you the output from start that you can navigate through using the arrow up/down keys or space (page by page). Press “q” to exit.

Extract text out of a image/PDF


For this purpose, you may read my previous related post here.

I am going to introduce (again) to the tesseract OCR engine. But this time I am using 16.04 and the command to install it is:

sudo apt install tesseract-ocr

If you have some PDF and want it to convert to image to further process it. You may use various methods. One of them may be:

convert input.pdf output.png

But this will produce a relatively low-resolution image that may result in bad text out of OCR.

So, instead we use:

convert -density 300 -quality 100 input.pdf output.png

Changing the density and tell it to not to decrease the quality than 100%.

Note if the input.pdf is a multi-page PDF, it will create different output images named like: output-0.png, output-1.png and so on.

So finally, use tesseract as:

tesseract output.png text_file -l eng

It will create a text_file.txt in the same directory. You may play with various options of convert or tesseract based on your needs.

Pandoc: convert document formats


I had to submit some text to mediawiki. As it supports special markdown, but I had a file on Google Docs. Initially, I looked up for some online tools for conversion. For them I downloaded the Doc in html format. But that didn’t work. For example: http://pandoc.org/try. It doesn’t support docx as input. But it support HTML. But it didn’t like much the HTML produced from Google Doc.

The thing that finally worked for me was that I installed pandoc on my system and then did the conversion.

Ubuntu guys can install is simply using:

sudo apt-get install pandoc

Then I read the manpage and found the -t flag useful. I searched for “mediawiki” in the manpage and that did the job.

The actual command that I fired was:

pandoc -t mediawiki input.docx > output

Then there will be a file named output will be created in the current directory in mediawiki style markdown format.

Update: More about pandoc can be seen at http://pandoc.org/ and the formatting help for mediawiki content can be found at https://www.mediawiki.org/wiki/Help:Formatting

IRC: Nickname is already in use


IRC is Internet Relay Chat. And it is a text-based protocol used for communication. It follows client/server architecture. There is a channel hosted somewhere on a server and you use your client application to connect to that server (channel).

I have used XChat IRC client and weechat (console based, for 2-3 times).

So coming back to the title of this post. Ah! You are probably here due to that only.

I had XChat set up on one operating system (let’s say Arch) and a nick registered e.g. mandeep_7. Now suppose, due to some reason, when I temporarily shifted to another operating system (let’s say Ubuntu). So if now I want to use the same Nick (/nick mandeep_7) then it will give the error: Nickname is already in use. There may be several reasons like sudden shutdown and you still are logged in to the particular server.

So what I tried is:

/ghost mandeep_7 password

Use your password instead of “password”. Now try the command /nick mandeep_7

You’ll now be able to identify yourself as old user (mandeep_7) on the new OS.

You should now try:  /msg NickServ identify <password>

Another thing to look for is linking.

Sourcehttp://archive.oreilly.com/pub/h/1940