Deploying OpenStreetMaps


Hey everyone,

OpenStreetMap is the map of the world with open license and freedom. I deployed OpenStreetMaps on my system after seeing a lot of people having a lot of troubles deploying them. I used Ubuntu 16.04, i3 (2-cores, 4-threads), 4GB RAM system for this and this link as source: https://www.linuxbabe.com/linux-server/openstreetmap-tile-server-ubuntu-16-04.

You need to download map file for your region of interest. I download India’s map (~400MB pbf file). You can download from here and of course can choose another region. Now let’s continue as the tutorial source I mentioned takes us. I am not going to repeat all of that (DRY), it’s easy. After installing some dependencies and postgresql, I created osm user, created db, created extensions.

Download Map Stylesheet from https://github.com/gravitystorm/openstreetmap-carto/releases and download the latest version which at this time is v4.0.0.

Next, you are going to import the download map file into the PostgreSQL database. If you have a small amount of RAM, then it’s recommended to create a swap space to speed up the import process. I created 2GB for me with the fallocate command. I noticed that fallocate doesn’t support NTFS file system (and a few other too). I tried creating on ext4 file system on another drive and it works fine. Then we mount it as swap memory. You can check it using: cat /proc/swaps or top commands.

Now after installing the osm2pgsql tool, the actual importing process starts. Following command does it all:

osm2pgsql --slim -d gis -C 3600 --hstore -S openstreetmap-carto-2.41.0/openstreetmap-carto.style great-britain-latest.osm.pbf

3600 (3.6GB) here is the amount memory you want to allow the import process to be taken. You can try taking a little more. I tried the double of it and it gave error. -d is for database name (gis). Change the files and folders according to your own system e.g. openstreetmap-carto-2.41.0 will probably be different on your system and great-britain-latest.osm.pbf will be name/path to your map file you downloaded. You can also add additional argument –number-processes=4. I added because of the 4 threads of my CPU. Hence the above command in my case becomes:

osm2pgsql --slim -d gis -C 3600 --number-processes=4 --hstore -S openstreetmap-carto-4.0.0/openstreetmap-carto.style india-latest.osm.pbf

It takes time to import. So, go out and enjoy life (plug in the charger/if on server, use tmux). Okay, it’s finished finally.

Now time to compile the mod_tile module required to serve the tiles. If you run into some error during the compilation, refer to this for installing some missing dependencies. After it’s done, let’s move forward.

Generate Mapnik Stylesheet

You’ll be asked to go to openstreetmap-carto-xxx directory and execute get-shapefiles script. In the newer version, they have converted the bash script to Python script and moved it to script directory. That’s why it’s recommended to follow the official guide. So go to script directory and execute it as:

python get-shapefiles.py

And then run in previous directory,

carto project.mml > style.xml

Above command (carto) gave me error:

carto: Unexpected token s

It was due to very old version of carto installed. Found the hint at https://github.com/gravitystorm/openstreetmap-carto/issues/2583 to use higher version (>0.16) of carto.

Installed latest version:

sudo npm install -g carto

You’ll need to install nodejs for this (npm – node package manager) to work. Then I executed carto again and it worked, despite giving some warnings like:

Warning: using the name attribute for layers (like building-text here) is deprecated and will be removed in 1.0.0. Use id instead.

But it’s OK. So if yours one is still giving the same error (carto), try logging out and log in again. Check if your carto version is updated. Mine shows:

$ carto -v
 carto 0.18.0 (Carto map stylesheet compiler)

Move on to configuring the renderd and edit /usr/local/etc/renderd.conf and edit your details carefully as per your own system.

After configuring Apache, if you point yourself to the browser at http://localhost/osm_tiles/0/0/0.png (or whatever ip) and get a image showing a world map, then you are lucky. If you did’t, you probably have to check Apache logs (/var/log/apache2/error.log) and hence this makes you more lucky. In my case, the tail of log file contains:

/var/run/renderd/renderd.sock No such file or directory

So, I checked and there wasn’t renderd directory. So I created on and given appropriate permissions. Then I executed again and I got the map image. Congratulations!

Display Your Tiled Web Map

Now, it’s time to see the map zoom-able and span-able. There are two JavaScript libraries available: OpenLayer and Leaflet. You  just need to download and extract it to your root of your web server like (/var/www/html). Edit the index.html file and put the OpenLayer specific code. Make sure to edit lines like: http://your-ip/ol.css. Replace your-ip with your ip-address. If you are testing on your local system, then put localhost instead. Point your browser to localhost and you should see the map. Now the tiles are generated on-the-fly. If you want to pre-render the tiles so that your processor doesn’t do processing everytime you zoom-in then you can do it via:

render_list -m default -a -z 0 -Z 10

This command will take some time. And yes, you can stop and resume it anytime. It won’t start from start again. It will generate tiles for various zoom levels.

The issue I was having at this step was no map was being shown. Seeing the apache logs disclosed the error:

socket connect failed for: /var/renderd/renderd.sock with reason: Connection refused.

I tried starting the renderd:

sudo systemctl start renderd

If you have nginx installed and running, stop it and change ports to avoid conflict with apache. Then restart apache and renderd. It should solve it.

Thank you.

Auto-mount disks on Startup


sudo fdisk -l

See which disk you want to auto-mount.

Suppose you attached a USB-drive and it’s shown as “/dev/sdb1” in above command.

Its file-system will also be shown beside that. Match your type from System name and notice the corresponding Linux type.

 

System name

English name

Linux type

 

W95 FAT32

Microsoft FAT32

vfat

 

W95 FAT32 (LBA)

Microsoft FAT32

vfat

 

W95 FAT16 (LBA)

Microsoft FAT16

vfat

 

W95 Ext’d (LBA)

Microsoft extended partition

Not used

 

NTFS volume set

Microsoft NTFS

ntfs

 

NTFS volume set

Microsoft NTFS with read-write access

ntfs-3g

 

Apple_HFS

Apple HFS

hfsplus

Suppose it’s W95 FAT32 (LBA) so your Linux type is vfat. If you are unsure about this type, you may consider mounting the device using: sudo mount /dev/sdb1 and then run mount command to see mounted file-systems and it’s file-system type.

After gathering all the necessary information, open /etc/fstab with root permissions with your text editor e.g. sudo vim /etc/fstab

Then add the new entry,

/dev/sdb1       /mnt/usb_drive       vfat       errors=remount-ro       0      0

to fill in line with the following.

<file system>     <mount point>   <type>       <options>         <dump>  <pass>

Then save the file and exit. You may check on the next reboot if it is mounted.

errors=remount-ro option is for the case when some error occurs, the device will get remounted as read-only. The <dump> option specifies if the file-system needs to be dumped (back-up) for safe keeping. 0 here means it doesn’t need to be dumped. And the <pass> means that the order which filesystem checks are to be performed at system reboot. The root filesystem should be labeled as 1, while others as 20 if we don’t want it to be checked. See man fstab command for more.

See “less” output at a time


There are a lot commands/tools that produce a lot of output that we can’t see at once (and without scrolling). Most of the commands on terminal can be made to show us whole output page by page using “less” command.

less command_here

less ls -R ~/

or you may pipe the output to less command,

ls -R ~/ | less

 

or similarly, to see the contents of a file:

cat file_name | less

There are many commands/tools that have their own prompt.

 

For IPython, you may do:

%page variable_name

 

Similarly for MySQL,

pager less -SFX

Or you may also try:

SELECT * FROM sometable \G

 

Above commands will show you the output from start that you can navigate through using the arrow up/down keys or space (page by page). Press “q” to exit.

Vim: Add some text at the start/end of each line


Suppose, you have a file with 1000 lines. You want to add some text at the beginning/end of each line. You won’t think of doing it manually (actually, I can’t stop you). Here, I am telling you how to save you a lot of time.

For example, we have the following lines in the file:

Name

Email

Phone Number

Address

And you want your output to be:

“Name”,

“Email”,

“Phone Number”,

“Address”,

and so on.

In Vim, you can simply type:

:%s/$/",/

It will place “, at the end of each line.

Similarly, for beginning:

:%s/^/"/

It will place ” at the beginning of each line.

Hence, we used $ for appending to end and ^ for prepending.

You can also do this for a few lines by visually selecting them with and then pressing : and type the command

s/^/"/

Skip the “%” because we do it while doing things globally across the file. So it will look like:

:'<,'>s/^/"/

Another method, if you want to type some text interactively on a couple of lines:

Ctrl+V

Select a few rows (downside, pressing j). After selecting the position in couple in all rows, press:

Shift + i

They type what you want. Then press the Esc key:

Esc

Now, wait 1 second. You’ll see same text to be replicated over all rows you selected.

Please share in comments, if you even easier method to do the similar.

Extract text out of a image/PDF


For this purpose, you may read my previous related post here.

I am going to introduce (again) to the tesseract OCR engine. But this time I am using 16.04 and the command to install it is:

sudo apt install tesseract-ocr

If you have some PDF and want it to convert to image to further process it. You may use various methods. One of them may be:

convert input.pdf output.png

But this will produce a relatively low-resolution image that may result in bad text out of OCR.

So, instead we use:

convert -density 300 -quality 100 input.pdf output.png

Changing the density and tell it to not to decrease the quality than 100%.

Note if the input.pdf is a multi-page PDF, it will create different output images named like: output-0.png, output-1.png and so on.

So finally, use tesseract as:

tesseract output.png text_file -l eng

It will create a text_file.txt in the same directory. You may play with various options of convert or tesseract based on your needs.

adb not detecting the phone!


What is it?

adb is abbreviated for “Android Debug Bridge”. It comes with the Android Development SDK but it can be installed separately too. It allows users to send commands to the Android phone via PC. It will be helpful in cases if you are tinkering with your phone and reached some nearly bricked condition.

In my case: I have a Zuk Z1 and I have installed TWRP recovery and Cyanogen 13 installed as of now. There might be some case or something went wrong that I can’t boot into my phone and even recovery isn’t working. Then booting the phone into bootloader might help. But I didn’t find any way to boot into bootloader without going into recovery. And suppose recovery isn’t working too. Then adb can help you here. You may use adb to boot into bootloader and then flash recovery and the OS later on.

Installation

On debian based systems e.g. Ubuntu, you may install it like:

sudo apt-get install android-tools-adb android-tools-fastboot

If it didn’t work, then see here. It will get you adb and fastboot.

My Problem:

The problem was that after installing adb, I connected the phone and fired the command:

adb devices

but it wasn’t listing my device.

So after searching a bit, I got a solution.

First you need to know the vendor id of your device. Search the Internet for that.

Other thing you can try is connect your phone to Linux PC and from terminal type:

lsusb -v

and look for your device and then look for “idVendor” field. For example mine is: 0x2b4c

Another thing you may try is to use following command from terminal:

fastboot devices

after booting the phone into the bootloader.

So coming back to my problem, my device wasn’t getting listed on running adb devices.

The solution is that you need to create/edit a file ~/.android/adb_usb.ini and write it as:

# ANDROID 3RD PARTY USB VENDOR ID LIST — DO NOT EDIT.
# USE ‘android update adb’ TO GENERATE.
# 1 USB VENDOR ID PER LINE.
# ie 0x2207
0x2b4c

And place the vendor id at the end, like I did above. Save file and run adb devices again. It will list the device but it will be unauthorized. Now your phone should pop up a permission box to allow the connection. Allow it (tick always).

Then do the following to restart the adb server (with sudo).

adb kill-server

and afterwards

adb start-server

Try reconnecting the phone after disconnecting if it didn’t work.

Import string as Dictionary in Python


I had a Python dictionary (that looks like json) stored in a file, generated by some code. Now I wanted it in Python to act as the dictionary. But when we read that file, it gets stored as string (in a variable). But I wanted to access its keys and values. But if it’s a string we couldn’t do this directly.

So I searched for a method to convert string to dict. Here is the go:

In [1]: import ast

In [2]: string = “{‘name’: ‘xyz’, ‘age’: 3}”

In [3]: string
Out[3]: “{‘name’: ‘xyz’, ‘age’: 3}”

In [4]: ast.literal_eval(string)
Out[4]: {‘age’: 3, ‘name’: ‘xyz’}

In [5]: the_dict = ast.literal_eval(string)

In [6]: the_dict.keys()
Out[6]: [‘age’, ‘name’]