SMS Gateway with gnokii

So, gnokii, what is that? As the authors sees it its just a name of a project aimed to develop tools and drivers for Nokia mobile phones. Tools and drivers which enables you to use your connected phone from your Linux system. Its all kind of like the Nokia data suite, which is shipped with more advanced models from Nokia, you can send sms, edit contacts and so on, actullay everything you do with your thumb normally. However gnokii is not really all that, but with the x-application xgnokii it is, but thats beyound the scope of this article. What we are interested in is a part of the gnokii package called smsd or sms daemon, which gives as rapid access to the phones sms capabilities. With the sms daemon up and running we can use php to interact with the phone, send and recieve sms, and also act differently depending on the content of the message, this is what we really want to do. The goal is to configure software and hardware so we can get the same kind of service as you would normally get from a big company selling mobile services like sms-gateways and so on, but with a minimal pricetag.

The final application that we will create throughout this article is built by 3 major parts, that all will need to be working(!).

1. The phone must be connected properly with the server.

2. The gnokii package with smsd must be configured correctly.

3. The phpscripts running the show must be written properly.

The flow of the application will be something as follows:

1. User send sms to the phone
2. Smsd picks it up and automatically puts it into its database.
3. Phpscript(smsparse.php / Listing ?) will scan smsds database for new messages.
4. Phpscript(smsparse.php / Listing ?) will examine the message.
5. Phpscript(smsparse.php / Listing ?) will act on the message, for example reply to the user with some sort of information.

I will further explain the differents parts in more detail later, but one thing at the time, and we’ll start from the top.

The bad thing about hardware is that it often costs a lot of money, but hopefully this project wont get you broke. What you’ll need in terms of hardware is a nokia phone and a cable for it so you can hook it up to your server. I will expect that you already have a server and its some sort of intel-like machine. In my environment I have used an Nokia 3310, which is quite new but is not a very expensive phone, but it’ll do for this. Normally there wouldnt be cables available for it, but a company from UK called Cellsavers(http://www.cellsavers.co.uk) have done something very genious, they sell a cable with some connector that you fit behind the battery on the phone. Very smart. For those that dont know, behind the battery, there is 4 metal pins, which is probably used by nokia to install software and other stuff on to the phone, but those Cellsavers managed to figure out how to use them, even on a budget phone like this. There might be other companies supplying the same thing, but I have not seen any around.

Another thing about the hardware, is to get a battery charger to the phone, it often comes one with the package. You can plug it in and leave the phone on forever, and not worry about the batteries. As seen on my picture, I have taken this a step ahead, I’ve used a usb charger for the phone. It just saves a power outlet in the server room.

Before starting the installation of gnokii and smsd, make sure you have mysql installed and working properly. Installing gnokii is quite straight forward, almost just the usual configure-make-make install. However there are some configuration options that I find important. The first might be a matter of taste, but I like to place everything belonging to gnokii be placed in /usr/local/gnokii . Therefore I will use –prefix=/usr/local/gnokii . Next one is the –without-x, since I will not use xgnokii to send sms and manage the phone, this is not needed, if you like to do that, skip this parameter, but it will add some more requirements to get working, look at the INSTALL file for that. Last one is the –enable-security, which will enable a lot of security features to the phone model of choice, like changing the pin code and so on. I find that good so I’ll include that.

The resulting configure line will be as follows:

./configure –prefix=/usr/local/gnokii –without-x –enable-security

So, lets get going, download the gnokii tarball from www.gnokii.org, the latest version currently is gnokii-0.5.5. First decompress the tarball and then we’ll start configure and make process.

# gzip -dc gnokii-0.5.5.tar.gz | tar -xof -
 
# cd gnokii-0.5.5
 
And then use the configure line we discussed above
 
# ./configure --prefix=/usr/local/gnokii --without-x --enable-security
 
If everything looks ok, continue
 
# make
 
and finally
 
# make install
 

If all output looked ok so far, you will have gnokii installed into /usr/local/gnokii . Before we test the phone, we will create the /etc/gnokiirc which will hold some configuration options as where the phone is connected and which model it is. My /etc/gnokiirc looks like this:

[global]
port = /dev/ttyS1
model = 3310
initlength = default
connection = serial
bindir = /usr/local/gnokii/sbin/

Make sure you connected your phone to the correct serial port as you specified in the configuration. Also check the model of your phone and enter it accordingly. The initlength variable controls the number of characters sent to the phone during initialization. Dont change this unless you have problems with the connection, go with the default initially.

The connection variable should be set to serial, if you’re interested its also possible to configure it to use an infrared connection.

Now, its time to test it all and see if you got it all right up to here:

# cd /usr/local/gnokii/
 
Try to send an sms with gnokii, you can send it to my gnokii phone and tell me you configured it all successfully.
 
# bin/gnokii --sendsms "+5555555555"
 
GNOKII Version 0.5.5
 
Please enter SMS text. End your input with <cr><control-D>:
 
Test 01 
 
Send succeeded!
 

If you get errors, check your settings in the /etc/gnokiirc .

Next step is to configure smsd, its important to have gnokii working before starting this, since smsd relies on the same runtime configuration and the library libgnokii. Get back to the directory where you uncompressed the gnokii tarball.

# cd gnokii-0.5.5/smsd/

Smsd can work either with a database or with a filesystem, but we will focus on configuring it using mysql as database that we mentioned earlier. Smsd is not compiled by default when you compile gnokii so this is what we will do now. Edit the Makefile and change the paths to the mysql installation in the DB Modules section.

# make

# make libmysql.so

# make install

If all output is ok, you should be able to continue setting smsd up under daemon tools. Otherwise, especially check your changes in the smsd Makefile to see that they are correct.

Since we want to use smsd with mysql we need to create a database and a user to it. Create a database in mysql called sms and a user with access to the database with username sms and password sms. For the example we will use simple usernames and passwords for simplicity, later you can change the password to something harder. Keep in mind that everyone that can access your sms database can insert rows into the outbox and therefore send sms from the connected phone. Worth to think about in bigger multiuser systems.

In the smsd directory is also a sql file called sms.tables.mysql.sql containing the table definitions. Import it into your database and you are all set to go. There is also a file for those that prefer PostgreSQL, but we will focus on MySQL here.

CREATE TABLE `inbox` (
 
  `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
 
  `number` varchar(20) NOT NULL DEFAULT '',
 
  `smsdate` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
 
  `insertdate` timestamp(14) NOT NULL,
 
  `text` varchar(160) DEFAULT NULL,
 
  `processed` tinyint(4) NOT NULL DEFAULT '0',
 
  PRIMARY KEY  (`id`)
 
) TYPE=MyISAM;
 
CREATE TABLE `actions` (
 
  `action_id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
 
  `trigger` varchar(20) NOT NULL DEFAULT '',
 
  `filename` varchar(200) NOT NULL DEFAULT '',
 
  PRIMARY KEY  (`action_id`)
 
) TYPE=MyISAM;
 
 
CREATE TABLE `smslog` (
 
  `log_id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
 
  `user` varchar(12) NOT NULL DEFAULT '',
 
  `sender` varchar(100) NOT NULL DEFAULT '',
 
  `message` varchar(255) NOT NULL DEFAULT '',
 
  `timestamp` int(11) NOT NULL DEFAULT '0',
 
  PRIMARY KEY  (`log_id`)
 
) TYPE=MyISAM;
 
 
CREATE TABLE `outbox` (
 
  `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
 
  `number` varchar(20) NOT NULL DEFAULT '',
 
  `processed_date` timestamp(14) NOT NULL,
 
  `insertdate` timestamp(14) NOT NULL,
 
  `text` varchar(160) DEFAULT NULL,
 
  `processed` tinyint(4) NOT NULL DEFAULT '0',
 
  `error` tinyint(4) NOT NULL DEFAULT '-1',
 
  `dreport` tinyint(4) NOT NULL DEFAULT '0',
 
  PRIMARY KEY  (`id`)
 
) TYPE=MyISAM;
 

Installing daomontools is quite straight forward, not much options or configuration directives, just stick to the normal procedure, and it have never failed for me. Well, almost, there is one thing to remember, caused by some differences in newer glibc(2.3.1 and above) versions you might need to patch the daemontools source before you try to compile it. The pathch is called the “errno-patch” and fixes some incompatible errno declaration made in the source. Some say this is due to bad practice in the programming, but the error really occurred when changes where made to the glibc software, so I dont know. Whatever actually, patch it and it works great.

If you need the patch, get it from http://www.qmail.org/moni.csi.hu/pub/glibc-2.3.1/ where patches for all DJBs software is available. Then, follow the daemontools installation instruction which is found at http://cr.yp.to/daemontools/install.html .

If you’re not familiar with patching software, this is done by downloading the software, and extracting it, and then use the patch program to do the final work. More information about the errno patching process and daemontools can be found at http://www.qmail.org/moni.csi.hu/pub/glibc-2.3.1/INSTRUCTIONS .

# tar zxvf daemontools-0.76.tar.gz
 
# cd admin/daemontools-0.76
 
# patch -p1 /path/to/daemontools-0.76.errno.patch

The phpscript we will be doing will be a small-as-possible one, since it will be running all the time on your sms-server. It will continuesly check if there are new messages in the sms inbox and if so, match them against the possible keywords that we have created.

We will call the script smsparse.php(LISTING ?) throughout the article. We will also create a few keyword scripts, they will be created as standalone phpfiles in a subdirectory called keywords.

The directorystructure and filestructure will look like this, when we have one keywordscript, with the keyword “hello”.

	./keywords/

	./keywords/hello.php

	./smsparse.php

Each keyword will, as you can see, have its own phpfile, and the name of the file will be the very keyword we wish to use. More on the content of the keywordfile later. Lets have a look at smsparse.php(LISTING ?), in the beginning of the script we will read through whats in the keywords directory, this is done in read_keywords() which is called on line 146. Each file that it finds in the directory will be matched against the ereg pattern on line 41, if it matches we will create an array with the keyword as a key, and the path to the file as value. This will be returned at the end.

Back at line 148, we will sort the array backwards after its key, this is done, since a longer keyword should match a message before a shorter keyword. Lets say we get a message with the following content “High fidelity”, and have 2 active keywords, one is “hi” and the other one is “high”. Every message should only be matched by one keyword, and if we check the message against “hi” first, it will match and the “high” will never be able to get it, so, a longer keyword should always be tested first, so in this case “high” would match, and then no more match will be tried.

Well, lets move forward to the main part of the application, on line 163(change) we will start a loop with 100 iterations, in each iteration we will check if new messages have arrived to the inbox, if so match the message against the active keywords as described above, and finally sleep for 1 second and then continue to the next iteration. When the message is matched against keywords, it will also be marked as processed in the database.

For the matching process of the script I have made two alternatives to show different approaches. The first one(LISTING ?, line 74-92) is the most fault tolerant but also the slowest one. The second one( LISTING ?, line 98-118) is not as tolerant to user mistakes but will instead save some computing resources. The difference will probably not be so dramatic, but on a heavily loaded server or poor hardware it will do the difference. You should figure out whats most important to you when you decide which approach is best for you, do you have a lot of computing power or do you have clever users.

Alternative 1 is the match_message() function(LISTING ?, line 74-92), the message is first cleaned of unwanted characters such as non-alphanumerics and spaces, it is also converted to lowercase on line 77. We will then go through all keywords and do an ereg() match against the “clean” version of the message. If the ereg matches we will include the keyword file and call the keyword function.

Alternative 2 is the match_message_fast() function(LISTING ?, line 98-118). Instead of the slower ereg() method, we will take the first word(LISTING ?, line 101-14) in the message and convert it to lowercase(LISTING ?, line 106). The word extracted from the message will then be looked up in the keywords array(LISTING ?, line 108). If a match is found, we will do the same again, include the keyword file and call the keyword function.

Heres the collection of phpfiles, its poorly documented, and the listingreferers are not up to date, but you should find your way around: Smsparse php files 0.1

The keywordscripts contains one or more functions who describes what should happen when a message is matched by a keyword. In our example we match the word “hello” in the beginning of a message. We well then reply with a “Hello to you my friend!”, this is shown in hello.php, LISTING ?. The function in hello.php is named hello(), the smsparse will try to call a function with the same name as the keyword(see LISTING ?, line 109-110 or line 82-83), if the function exists, it will also pass the message and the senders phonenumber as arguments.

Sending the reply to the sender by sms is a simple process of adding a line into the outbox table(see LISTING ?, line 11). The sms daemon will then poll the table for new messages and send them.

The configuration of daemontools is a bit special compared to other things, and there is a bunch of files and directories, however, after a while it seems quite organized and easy to work with. To start with, when daemontools is installed, it creates a directory called /service . The /service directory will contain all services that daemontools is running. Actually there is a program within daemontools called supervise that monitors the /service directory and takes care of starting and keeping the services running. Compared to “normal daemons” which are started at boottime, those services are started by supervise, and if you kill any of the services or they die by themselves, supervise will take care of starting them again for you. So actually, its a good thing to use when you want services running all the time, no need for extra monitoring stuff. However, not all services are suitable to run under supervise, they have to behave in a certain manner, but most can be configured and changed to work.

But, on to the configration of our smsparser under supervise. We will assume that you have a php command line interpreter installed, you can usually check this by entering:

# whereis php

On my machine, a red hat 8, with php installed it returns this:

php: /usr/local/bin/php /usr/local/lib/php /usr/local/lib/php.ini

This means that I have the php binary in /usr/local/bin/php . This is important to know later on. We will assume that the smsparse.php and the keywords/ directory described above, is located at /usr/local/smsparse/ . We will create a service directory for smsparse in the /usr/local/smsparse/ directory, we will call the directory supervise-smsparse .

# mkdir -p /usr/local/smsparse/supervise-smsparse/

The smsparse service directory will contain all information that supervise needs to run smsparse correctly. To begin with, we will focus on getting smsparse running, and then we will add the loggin functionality.

In the /usr/local/smsparse/supervise-smsparse/ directory, we will create a file called run containing the following:

#!/bin/sh
exec /usr/local/bin/php -q /usr/local/smsparse/smsparse.php

As you remember from the code, the smsparse.php will only run for 100 iterations(line 163 change) and then exit. Supervise will than make sure that its started again, this will somewhat prevent the php process to grow in memory over time.

Just by creating this directory, supervise will yet start it, we will have to link it into the /service directory with a symlink first.

# ln -s /usr/local/smsparse/supervise-smsparse/ /service/supervise-smsparse/

We will do the same for smsd so we will have supervise monitor the smsd process as well. As described earlier we installed gnokii in /usr/local/gnokii/ so the smsd binary will reside in /usr/local/gnokii/bin/smsd . We will create a subdirectory to to house the run files for smsd as we did to smsparse

# mkdir -p /usr/local/gnokii/supervise-smsd/

Another run file will be created at /usr/local/gnokii/supervise-smsd/run that contains:

#!/bin/sh
exec /usr/local/gnokii/bin/smsd -u sms -p sms -d sms -m mysql

The parameters are the username, password and the database which you created for smsd, be sure to replace those with the ones you will use. To start the smsds run file, we link the supervise-smsd directory into /service with:

# ln -s /usr/local/gnokii/supervise-smsd/ /service/supervise-smsd/

If you now check your processlist, you will see your smsparse and smsd process listed if you have done everything right. Both of the processes are now running and hopefully doing their job.

# ps axf

Results in something like this on my machine, non-interesting parts are left out:

  776 ?        S      0:00 /bin/sh /command/svscanboot
  805 ?        S      0:01  \_ svscan /service
  819 ?        S      0:00      \_ supervise smsd
 3009 ?        S      0:00      |   \_ /usr/local/gnokii/bin/smsd -u xxx -p xxx -d xxx -m mysql
  820 ?        S      6:26      \_ supervise smsparse
23813 ?        S      0:00          \_ /usr/local/bin/php -q /opt/www/smsparse/smsparse.php

As for the smsparse process we would also like to have some logging features. Together with daemontools comes a program called multilog which can handle a directory with a set of log files and rotate them automatically. So, when using multilog there are no need to write special log rotating scripts, multilog takes care of it all.

To add the logging functionality, start out by creating a log directory in supervise-smsparse by

# mkdir -p /usr/local/smsparse/supervise-smsparse/log

The logging process acts quite much like a normal process running under supervise, it has its own directory and run file, so create a special run file at /usr/local/smsparse/supervise-smsparse/log/run that contains:

#!/bin/sh
exec multilog t ./main

Multilog accepts any number of arguments where each argument specifies an action. You can create actions that just includes certain lines in the log and other nice features. Our line above however is quite simple, and really just says that “add a timestamp on each line, and store the logfiles in ./main”. The t argument represents a precise timestamp i.e the number of TAI seconds since 1970-01-01 00:00:10 TAI. As you might remember from the smsparse.php script, we included a date(’Y-m-d H:i:s’) before each output, so actually we will have double timestamps in the log file so you really could skip the t argument, but its good to learn about, so we’ll just keep it.

Now, the log directory will not be linked directly to the /service directory, instead supervise starts it automatically since its in an already linked directory. But, you must restart the supervise daemon to make it aware of the log directory, so use the svc program to send a TERM signal to the service.

# svc -t /service/supervise-smsparse/

A new check at the processlist will show you that smsparse is started again together with the logging process.

  776 ?        S      0:00 /bin/sh /command/svscanboot
  805 ?        S      0:01  \_ svscan /service
  819 ?        S      0:00      \_ supervise smsd
 3009 ?        S      0:00      |   \_ /usr/local/gnokii/bin/smsd -u xxx -p xxx -d xxx -m mysql
  820 ?        S      6:26      \_ supervise smsparse
23813 ?        S      0:00      |   \_ /usr/local/bin/php -q /opt/www/smsparse/smsparse.php
  818 ?        S      0:00      \_ supervise log
  837 ?        S      0:01          \_ multilog t ./main

This is good, your services are now supervised by supervise and will run forever, you even have a log on smsparse to make sure it doesnt do any mistakes. To monitori the log, you can use the usual tail program, try that with:

# tail /service/supervise-smsparse/log/main/current
@400000003ffc1dfc22cb14e4 2004-01-07 15:54:05: Starting sms parser...
@400000003ffc1dfc22cb2484 2004-01-07 15:54:05: hello
@400000003ffc1dfc22cb2c54 2004-01-07 15:54:05: success

This will show you that the smsparser was started correctly, and 2 keywords where found, the hello and success keywords. At the beginning of the line, the TAI timestamp is showing. To translate the timestamp to a human readable form, you can pipe the output through tai64nlocal like this:

# tail /service/supervise-smsparse/log/main/current | tai64nlocal 
2004-01-07 15:57:27.380601500 2004-01-07 15:55:46: Starting sms parser...
2004-01-07 15:57:27.380605500 2004-01-07 15:55:46: hello
2004-01-07 15:57:27.380607500 2004-01-07 15:55:46: success

Notice the difference of the timestamps, the first one is about 2 minutes later than the second one, this is because the leftmost timestamp is taken when multilog receives the line and the rightmost when it actually happened. Php seems to hold the information until the loop has ended and then release it to stdout. The use of flush() and similar approaches was not helping, but its not an issue here.

To test your new fancy setup of gnokii and your php parser, grab another cellular and send an sms containing “Hello” to your gnokii phone and hopefully if everything is set up right you will get a message with the message we entered in the hello keyword script.

Smsparse php files 0.1

As you might have noticed throughout the article, its not that hard to set up a mobile service where you gather information from users sms messages. Lets say you want a voting system somewhere where you do not have a computer. You can ley people send sms with their vote, they will pay the fee for the sms and as long as you’re not sending one back, this application will not cost anything to you. The result could then be displayed on a webpage or any other media of your choice.

The list of possibilities can be long, and I have found several areas where this can come handy. Clients have shown great interest in using this as an complement to other services.

If you’re one of those that are worried about scalability it actually is an issue, this soloution will probably not fit the organization that want to push out thousands of sms every day. But, this might be a good place to start, to check the terrain and interest a bit, if your customers appreciate it, you might want to advance further to one of the more advanced sms services that some companies offer. Most of them will probably give you at least the same functionality or even more.

However you can probably stay with gnokii for quite some time, newer versions of gnokii supports the use of multiple phones.

I like gnokii since its a simple and cost effective way to get a very cool extra feature which I personally use a lot and some of my clients use daily for various tasks.

Note: This article where written in 2004, but never got published as intended, and due to a few questions about the subject its published here instead of the original location.