September 20, 2006

qmail: 7 day mail queues? too long.

i've been taking a proactive stance in checking the mail queue in my office, since if it gets cluttered with newsletters or unnecessary stuff (including the occasional password phishing from code vulnerabilities in contact forms), it ends up slowing down other emails significantly.

by default, the qmail queue is 7 days long (604800 seconds). to check that, you can run the following:

# qmail-showctl | grep queue
queuelifetime: (Default.) Message lifetime in the queue is 604800 seconds.

(side point: there's a lot of cool stuff you can see there related to the qmail setup if you don't only grep for the queue.)

in my opinion, 7 days is just way too long. sometimes i'm checking the queue and an email is mailed to a wrong address... and the email just sits there while the mailserver repeatedly attempts to send the message to this nonexistent address. (for example, if you're looking to email and you accidentally addressed it with the domain, you'll be waiting a long time for a bounceback, which might cause frustration and anger because you thought you sent it to the right guy to begin with.)

everything on linux can be tweaked, and it's relatively easy to do at times. in this particular case, what is needed is a newly created file, /var/qmail/control/queuelifetime, which contains a single line: the number of seconds that you want the queue to last. in my case, i made it 172800 seconds (2 full days; a single day is 86400), so these emails get returned to sender informing them that they should get the right address or try later.

once you run this file, you can verify that the new queue length is in effect by running the following:

# /var/qmail/bin/qmail-showctl | grep queue
queuelifetime: Message lifetime in the queue is 172800 seconds.

note how it doesn't say "Default" anymore like the previous execution of the same command did.

to force those old emails to be sent? just run qmHandle -a and you'll notice that the queue (qmHandle -l) has gotten a lot shorter.

if you don't have qmHandle, you can get it on sourceforge; just click here. it's not part of the regular qmail distribution. more information on qmHandle can be found in this blog entry.

September 19, 2006

robots.txt and spidering.

when you have content that is not for public consumption, you should always be safe than sorry by preventing the search engines from crawling (or spidering) the page and learning your link structure. for example, in a development environment, it would hardly be useful for the page to be viewed as if it's a public site when it's not ready yet.

enter robots.txt. this file is extremely important; search engines look for that file and determine whether the site can be entered into its search cache or if you want to keep it private.

the basic robots.txt file works like this: you stick the file in the root of your website (e.g. the public_html or httpdocs folder. it won't work if it's located anywhere else or in a subdirectory of the site.

the crux of the robots.txt is the User-Agent and disallow directives. if you don't want any search engine bots to spider your any files on your site, the basic file looks like this:

User-agent: *
Disallow: /

however, if you don't want the search engines to crawl a specific folder, e.g., you would create the file as so:

User-agent: *
Disallow: /private/

if you don't want google to spider a specific folder called /newsletters/, then you would use the following:

User-agent: googlebot
Disallow: /newsletters/

there are hundreds of bots that you'd need to consider, but the main ones are probably google (googlebot), yahoo (yahoo-slurp), and msn (msnbot).

you can also target multiple user-agents in a robots.txt file that looks like this:

User-agent: *
Disallow: /
User-agent: googlebot
Disallow: /cgi-bin/
Disallow: /private/

there's a great reference on user agents on wikipedia. another great resource is this robots.txt file generator.

where security is concerned, a robots.txt file makes a huge difference.

September 18, 2006

showing and understanding mysql processes in detail.

i've learned a little trick on how to determine how your mysql server is running and where to pinpoint problems in the event of a heavy load. this is useful in determining how you might want to proceed in terms of mysql optimization.

# mysql -u [adminuser] -p
mysql> show processlist;

granted, on a server with heavy volume, you might see hundreds of rows and it will scroll off the screen. here are the key elements to the processlist table: Id, User, Host, db, Command, Time, State, Info, where:
Id is the connection identifier
User is the mysql user who issued the statement
Host is the hostname of the client issuing the statement. this will be localhost in almost all cases unless you are executing commands on a remote server.
db is the database being used for the particular mysql statement or query.
Command can be one of many different commands issued in the particular query. the most common occurrence on a webserver is "Sleep," which means that the particular database connection is waiting for new directions or a new statement.
Time is the delay between the original time of execution of the statement and the time the processlist is viewed
State is an action, event, or state of the specific mysql command and can be one of hundreds of different values.
Info will show the actual statement being run in that instance

another useful command is:
mysql> show full processlist;
which is equivalent to:
mysqladmin -u [adminuser] -p processlist;

this shows my specific query as:
| 4342233 | adminusername | localhost | NULL | Query | 0 | NULL | show full processlist |

or you can display each field in a row format (vertical format), like so, simply by appending \G to the end of the query:
mysql> show full processlist\G

this list is very likely preferable in the event that your data scrolls off the screen and you want to find out the specific field name of a value in your database.
******** 55. row ********
Id: 4342233
User: adminusername
Host: localhost
db: NULL
Command: Query
Time: 0
State: NULL
Info: show full processlist

you can also check how many mysql queries a user has open by running the following command:
mysqladmin -u [adminuser] -p pr | awk -F\| {'print $3'} | sort -nk1 | awk -F_ {'print $1'} |uniq -c |sort

to see which database has the most active queries, run the following:
mysqladmin -u [adminuser] -p pr | awk -F\| {'print $3'} | sort -nk1 |uniq -c |sort

oh, and since it's useful... here's a recommend /etc/my.cnf:

slave_net_timeout = 50
delayed_insert_timeout = 50








another fine tuning would include the following and is good for machines with plesk:

key_buffer = 128M
max_allowed_packet = 1M
table_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 64M
thread_concurrency = 8

the above will help you optimize your mysql database as well, but the configuration isn't for everyone.

August 22, 2006

resolving canonical issues with plesk.

in the world of SEO (search engine optimization), there is an unwritten rule (well, it will be written sooner or later) that you can't have duplicate content on google search engines from the same site. this means that and cannot both be found by search engines. you must choose one or the other or you may face a penalty.

there's an easy solution for this using vhosts in plesk. the only not-so-user-friendly part about this that you have to do it for every domain you are worried about, and with 100+ domains, you'll be making 100+ (or 200+ files if you have SSL support as well) vhost files for each domain.

in any event, this is how it's done.

navigate on your plesk server to your domain's conf directory. on some machines, it's

# cd /var/www/vhosts/

i prefer going through this shortcut:

# cd /home/httpd/vhosts/

regardless, both are symbolically linked -- or they should be in certain setups.

create the file vhost.conf

# vi vhost.conf

add the following to the vhost.conf file

RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}$1 [QSA,R=301,L]

for domains with SSL support, you will need to create a file called vhost_ssl.conf as well.

# vi vhost_ssl.conf
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ https://www.%{HTTP_HOST}$1 [QSA,R=301,L]

that's it! now, run this plesk command to process your update.

# /usr/local/psa/admin/bin/websrvmng -av

load your page in your preferred web browser as it will automatically redirect to and will be reflected in search engines with the www prefix only.

July 28, 2006

finding files with specific permissions.

in may, we did a plesk migration from an older server to a newer one. it was relatively smooth, but for some reason, a lot of image permissions were not kept intact (instead being read+write only by the owner, but not the user or group).

i am beginning to learn how useful the find command is. this code snippet below enabled me to find all of the jpg images on my server that were chmodded 600 ... and made them appropriately readable.

# find /home/httpd/vhosts -name '*.jpg' -perm 600

but wait, you can execute commands on these results too!

# find /hme/httpd/vhosts/ -name '*.jpg' -perm 600 -exec chmod 644 {} \;

good stuff!

July 11, 2006

iptables port redirection (smtp servers).

this is a quick tutorial on tables port redirection.

we have a few clients whose outgoing port 25 is blocked by their ISP due to vulnerabilities and exploits that cause this port to send out considerable amounts of spam.

this is never an obstacle for someone who hosts on a linux server with iptables, because you simply can forward traffic from port 25 to another port (e.g. 26, as in the example below):

# iptables -t nat -I PREROUTING -p tcp --dport 26 -j REDIRECT --to-ports 25

easy, isn't it?

of course, if 26 is blocked, you can substitute another open port -- but 25 must remain the same, as 25 is the standard smtp (outgoing email) port.

for someone who needs to utilize these settings, s/he would be required to go to outlook or whatever email application is being used and change the smtp port from standard 25 to 26 (in outlook, this can be done by going to tools > email accounts > view or change existing email accounts > select the questionable email account > change > more settings > advanced tab > change 25 to 26).

then, try to send out the email. it should no longer be sitting in the user's outbox.

to verify that the change was made on the linux side, simply run:

# iptables -t nat --list

the output should show something like this:

target prot opt source destination
REDIRECT tcp -- anywhere anywhere tcp dpt:26 redir ports 25

confirmed -- it is working!

make sure to save your changes so that the rules are maintained between reboots:

# /sbin/iptables-save

June 30, 2006

bcc using outlook 2003.

this is a little off-color here, given that the target audience of my blog is intended to be the linux-interested crowd, but this is something i really didn't know, so i figured this would be the perfect place to notate it.

outlook generally only displays the To and CC fields. what about if you need to send emails using BCC (blind carbon copy)? well, then you can simply press Ctrl+Shift+B to get the screen that displays the BCC field. then, type your addresses there.

that was easy.

June 29, 2006

backing up and restoring databases using phpMyAdmin.

i'm very much a command-line-interface kind of girl, but not everyone has shell access. hopefully, they do have phpMyAdmin, one of the most essential tools for managing your database.

the tutorial below gives you an idea of how to perform a database backup and restore using phpMyAdmin.

database backup.
1. open your phpMyAdmin webpage. hopefully, you have a link to it somewhere in your website's control panel.
2. click on the database name on the left-hand navigational bar (generally in black text, with a number in parentheses indicating the number of tables in the database)
3. click on Export on the top of the screen.
5. check the option next to Save as File
4. hit the Go button.
5. download the *.sql file.

you can open this file in Notepad or any text editor and see the database schema with all of its contents intact.

database restore.
1. open your phpMyAdmin page.
2. click the database name on the left-hand navigational bar.
3. assuming you're replacing the current database, you first need to drop all of the tables, so make sure to check all of the tables (the checkbox is immediately to the left of the Table column)
4. in the With selected: drop down box, choose Drop
5. now that you've emptied out your database, click on SQL on the top bar.
6. click the Browse button and locate the *.sql file that you created in your backup.
7. click Go.

you've now restored your database.

if you wanted to create an entirely new database with the same schema, simply open the *.sql file you created in your backup, and on the 7th line or so, you will see Database: `db_name`. just replace this with the new database name you wish to create.

June 27, 2006

mysql dump and restore.

i've seen a tremendous amount of different syntaxes for the most simplest database dumps and restore. why worry about the details? this is how you perform a database backup:

$ mysqldump -u username -p databasename > filename.sql

type your password at the prompt, wait a few seconds, and you now have a perfectly good .sql file that represents your database schema and all the data within.

to restore your database, use the following:

$ mysql -u username -p databasename < filename.sql

enter your password once again at the prompt. this input assumes that you have a file named filename.sql in the directory in which you are performing the backup. you can reference the /path/to/filename.sql as well.

that's all there is to it. if you need to change any data before the restore, simply edit the filename.sql with your favorite editor (but for your own sake, make sure it's not that large!)

simple backup scripts.

i think there's a golden rule in system administration: never underestimate the importance of backups. i have a few small scripts that you can use with rsync or just local that i decided may be useful, and i'm going to share them here. maybe later on when i master rsync, i'll post a tutorial about it here as well.

a mysql backup may work for you as follows:
mysqldump -A --password=xxxxxx | gzip > /root/mysql_backup/`date +%m%d%y_%T`.sql.gz

on the other hand, if you want to backup all qmail for a particular domain, you can attempt to do a job as such:

tar zcvf /root/domain/mailbackup-`date +%m%d%y_%T`.tar.gz /var/qmail/mailnames/

i have a cron job that runs nightly to execute these tasks, and i find them rather useful, especially since the inclusion of the timestamp gives me an idea of what could have happened and when.

June 26, 2006

qmail and its 30 second delay.

the installation of nagios left me stumped in one area. i realized that running check_smtp in /usr/local/nagios/libexec/ kept taking exactly 30 seconds before it worked, and since the check_smtp executable gives a CRITICAL - Socket timeout after 10 seconds, it was pretty essential to get it confirmed as working within that 10 second period.

this issue only seems to plague users of plesk.

the trick is actually to edit the following file:

# vi /etc/xinetd.d/smtp_psa

it is necessary to add -R right after the server_args directive.

e.g.: server_args = -R /usr/sbin/rblsmtpd -r [snip]

then, simply restart xinetd.

# /sbin/service xinetd restart

now, on your remote nagios box, try it:

$ /usr/local/nagios/libexec/check_smtp -H

the response time should be instantaneous:
SMTP OK - 0.113 sec. response time|time=0.112892s;;;0.000000

now, only if check_mysql compiled correctly on my server, i'd really have all my bases covered!

how to reset passwords using phpbb.

while we don't like using open source software (i suppose movabletype is possibly one of the few exceptions), there are always folks who love using phpbb -- and then there are folks who love exploiting it.

i decided today that it would be a good idea to disable one of our inactive boards that was getting a lot of hits from spam bots -- except that i couldn't find the login information anywhere.

fortunately, i found out how to reset the password with this nifty little trick that requires, at the mininum, phpMyAdmin (so much for what i say about open-source software).

here's how it's done:

1. ftp or log onto the server and navigate to your phpBB directory. download or jot down the values in config.php. (you can bypass this step if you use plesk.)
2. log onto your phpMyAdmin with the username and password that you have written down in step 1.
3. on the left hand navigational bar, click on the phpbb_users table -- this is where your username and passwords are stored.
4. click Browse on the top of your screen to view a list of registered users.
5. search for the account that needs its password reset (e.g. Administrator)
6. click Edit (the little pencil icon works too)
7. from the user_password row, click the Function pull-down menu and choose MD5 (your password is stored in an MD5 hash)
8. in the empty box to the right of MD5, replace the password with your new password in plaintext
9. click Go at the bottom of the page.
10. that's it! now log in with the new password.