cron/user: Permission denied

What caused this? Why is it here? How to resolve? Read more

This was caused because of a faulty CPanel setup, or a faulty server setup. Either way, permissions were not set appropriately for fixing this issue. Here’s how I got it fixed, and a brief explanation of HOW this happens, WHY it happens, and whatnot (ie: the tech crap)

This happens because the user did not have permission to write to the specific file that cron needs to change in the /var/spool/cron directory. There are two ways to change this:

1. Make the directory world writable. This is a security risk, because this would let anything write anything to this directory, without any sort of security. Bad idea, but if you want to do this, you can do this like so.
   

   chmod a+w /var/spool/cron

2. Fix the permissions on cron so that the user can execute it with suid priviledges. This will allow the binary to write to this specific file, but ONLY the binary.
   

   chmod 4775 /usr/bin/crontab

Obviously the second way is more secure, because it enforces strict rules as to what can be done, what can’t be done, and the like. This keeps things secured to a degree, while still allowing security measures to be in place. Of course, someone abusing the crontab binary is STILL possible, but that’s for another entry.

What is this going to be used for?

If you’re going to use your server for hosting, then you are going to be going through an uphill battle. Thankfully, this entry will help you at least somewhat keep things stable.
Now that we’ve established what you’re using it for, it’s time to secure the server. Seriously, security MUST be placed on any server, updated, and managed as if it were your own computer (it is, after all, you’re just paying someone to use it).

Step One , the firewall
There are a few schools of thought on this, and none are “wrong” per se, none are correct. You NEED to have a firewall (iptables) interface in your system though if you’re running ANY kind of Linux Server online. The firewall will dictate whom can access which ports, and set the rules forth. This is VERY important! In the web hosting industry, the two more popular ones are APF and CSF. Do your homework on each, especially the authors of each, before you download and install either! I would also (very) strongly suggest that you harden your tcpwrappers, part of which is shown here .

Step Two, the system
Ok, we’ve got a firewall installed, now let’s move on to the system. Here are a few tips and tricks that you might want to use when securing systems
A> Disable compilers for normal users
B> Disable wget, get, lynx, curl for normal users
C> Secure /tmp so that exes can’t be run from it directly (note: this will NOT stop perl exploits!!)
D> Disable ICMP
E> Remove unused stuff (cups, etc)

Step Three, the logs
Your system talks to you on a daily basis, do you know what it’s saying? Better yet, do you know how to interpret what it’s saying?
Have your critical logs forwarded to you. These can be done from logwatch , but that will also ignore day to day things such as alerts from your firewall, etc. Don’t just have them forwarded, but LOOK at them, see what your system is telling you!

Step Four, the software
Update your software, ALWAYS use protective patches like suhosin to secure your software, whenever possible, and whenever it won’t defeat the purpose of your server. Keep your software up to date!
A few things NOT to do (unless you like a lot of headaches, that is):
– Never disable php functionality. It’s there for a reason, and your client needs access to these things. Instead, use something like suhosin to properly secure and harden php!

– Never use phpsuexec/suphp . These are serious deterrents to real php programmers and developers. Why?

1: phpsuexec/suphp don’t like php values in .htaccess files. This means that your advanced programmer (such as myself) who places multiple sites under one directory, and uses htaccess to control each site and directory is screwed. This will also upset and alienate advanced programmers. You ARE running a hosting business, right? Do you WANT individuals complaining about not being able to use software?
2: phpsuexec/suphp HATE symlinks. While this isn’t (usually) going to be a problem, it will if you’re running, say, large media sites that should have data stored on another drive

– Never use grsecurity for your kernel, EVER! The Linux Kernel is the core of the OS, the most important part of the entire beast, yet, the most fragile. Putting something like grsecurity in there is just asking for trouble. Not to mention, it is a beast and a half to configure properly for your users to access.

– Never allow full ssh access, ever. Yes, I stress that, because there is no reason that anyone should ever need ssh access. With advanced, on the fly editors like Ultraedit, changes can be made to websites from within Windows itself, and ftp’ed over to the server. The only reason a user would need “ssh access” is to compile something, to run a program (daemon) on your server, or something else. There are no other reasons, and if they’re doing that, you’d best have some sort of proof that they’re legitimate.

Now, following the above examples is just a start to providing a usable, secured server. Of course, any admin with half their sense to them will tell you anything is hackable, as long as it is online, or has a keyboard input. Them’s the breaks though, really. The difference between MOST and what is recommended here? You’re going to end up with client usability with what is recommended here. With most, you’re seriously cutting off client usability.

As always, questions, thoughts, and ideas are welcomed. Spam is not,

Tom

Mysql, in it’s raw format is binary. What happens when you try to click on an .exe that is only partially downloaded? Oops, you can’t do that. The very same risk is there with mysql, only MORESO, because of what mysql is, and how it handles itself.

Firstly, mysql, on a properly setup server can handle hundreds of queries a second. I’ve had my own 2 servers going at 500+ each repeatedly. Now that’s not good on processing and whatnot, but hey, it’s not a completely BAD thing.
Secondly, mysql, when properly setup, locks tables on execution. So, if it has to so much as go to the bathroom on another table, it locks a table so nothing else can deal with it. Smart thinking there.
Thirdly, mysql is god. Seriously. Ok, maybe not GOD, but it is used in EVERYTHING from blogs (this one is powered by it) to forums (phpbb,smf,ipb,vbulletin, all use it)to you name it, they all use it.

Now, let’s just say that you’re running along and doing a backup at midnight, and at the same time Joe Bloe invites his 5 friends to his blog, and they all make a comment on different entries. Well, if you’re doing things the RIGHT way, nothing happens, maybe those entries aren’t made, but maybe they are copied. No harm, no foul, right? However, IF you’re doing things the wrong way, the entire blog can be seriously messed up, due to the binary structure of mysql.

Let’s say you have to restore from those backups, the very next day. What’s going to happen to Joe Bloe’s blog? If you did things properly, nothing at all. if you did things the way that others mention to do it (rsync), it can , in fact, be a VERY big (and messy) deal. Joe Bloe’s blog (at minimum) is not going to be right, and will need repairing, and that’s ONLY if you’re lucky. Typically, much more will break.

Let’s say you’re moving from server A to server B . Server A has mysql4 on it, while server B has mysql 5 on it. OOPS, your backups are going to screw the entire server over, literally.

So, that said and done, how do we do backups PROPERLY, so that they will work in (virtually) any scenario with minimal issues? That’s easy.
First, we need our script:

DATE=`date +%m%d%y`
THISNEWDATE=`date +%m%d%y-%T`
THISMONTH=`date +%b-%y`
BACKUP=/backup/sql
OLDBACKUPDIR=/backup/sql-archived
BACKMONTH=$BACKUP/$THISMONTH
BACKDEST=$BACKMONTH/$DATE
echo “SQL Backup Started : $THISNEWDATE”;
if [ ! -d $BACKMONTH ];then
tar jcpf $OLDBACKUPDIR/$THISMONTH.tar.bz2 $BACKUP
rm -rf $BACKUP
mkdir $BACKUP
mkdir $BACKMONTH
fi

if [ $BACKDEST ];then
rm -rf $BACKDEST
fi

if [ ! -d $BACKDEST ];then
mkdir $BACKDEST
fi

for var in `find /var/lib/mysql/ -type d | \
sed -e “s/\/var\/lib\/mysql\///”`; do
mysqldump –add-drop-table $var >> $BACKDEST/$var.sql
done
THISENDATE=`date +%m%d%y-%T`
echo “SQL Backup Ended : $THISENDATE”;

Now, 4 variables need to be changed there:
A> backdest needs to point to an actual backup destination
B> oldbackupdir needs to be changed to the directory you want your ARCHIVED backups sent to

SO, an explanation as to what this does to your srever, how it’s run (as root), and the like:

Firstly, this checks to see if the backup destination exists. If it doesn’t, it takes care of that.
Secondly, it checks to see if the backup destination / month exists. If it doesn’t, then it archives everything from the PREVIOUS month to your archived directory (what you created in B)
Thirdly, it then makes a backup of the entire database structure in .sql dump format, adding the necessary –add-drop-table format to this.

I can’t tell you how many times over the years that this has worked for me. It’s literally a lifesaver. SQL is so critical that you SHOULD back it up every day, while flat files are usually good with once a week (though I do every day as well), and incremental backups.

So, now that we have the script in a file, how to use it? We call it through cron:
From the shell (as root)

crontab -e

add this to the cron tab

0 1 * * * /path/to/sqlback.sh > /dev/null 2>&1

Good luck with your backups!

Tomorrow, how to create daily SQL backups OUTSIDE of your control panel, and restore from them if necessary!

First, we need a script. This script will be a php script, used to determine the databases, and whatnot. This is a php script, so create a .php file on your server (as root), and put these contents in there:

#!/usr/local/bin/php
$myuser=”root”;
$mypass=”insertyourrootpasswordhere”;
$myhost=”localhost”;
$query = “SHOW DATABASES”;
$mtime = microtime();
$mtime = explode(” “,$mtime);
$mtime = $mtime[1] + $mtime[0];
$start_time = $mtime;

$dbh=mysql_connect (”$myhost”, “$myuser”, “$mypass”) or die(mysql_error());
$result = mysql_query($query) or die (”Error in query: $query. “.mysql_error());
$count = 0;
while ($row = mysql_fetch_array($result)) {
$count = $count + 1;
$thisdb=$row[0];
cleanup($thisdb);

}
function cleanup($dbname)
{
check_exists($dbname);
global $myuser, $mypass, $myhost;
$dbh=mysql_connect (”$myhost”, “$myuser”, “$mypass”) or die(mysql_error());
mysql_select_db ($dbname) or die(mysql_error());
$db_clean = $dbname;
$tot_data = 0;
$tot_idx = 0;
$tot_all = 0;
$local_query = ‘SHOW TABLE STATUS FROM ‘.$dbname;
$result=mysql_query($local_query) or die(mysql_error());
if (mysql_num_rows($result)) {
while ($row = mysql_fetch_array($result)) {
$tot_data = $row['Data_length'];
$tot_idx = $row['Index_length'];
$total = $tot_data + $tot_idx;
$total = $total / 1024 ;
$total = round ($total,3);
$gain= $row['Data_free'];
$gain = $gain / 1024 ;
$total_gain += $gain;
$gain = round ($gain,3);
$local_query = ‘OPTIMIZE TABLE ‘.$row[0];
$resultat=mysql_query($local_query);
if ($gain == 0) {

} else {

}
}
}

while ($row = mysql_fetch_row($result)) {
$histo += $row[0];
$cpt += 1;
}
}
function check_exists($dbname)
{
global $myuser, $mypass, $myhost;
$dbh=mysql_connect (”$myhost”, “$myuser”, “$mypass”) or die(mysql_error());
mysql_select_db ($dbname) or die(mysql_error());
}
function microtime_diff($a, $b) {
list($a_dec, $a_sec) = explode(” “, $a);
list($b_dec, $b_sec) = explode(” “, $b);
return $b_sec – $a_sec + $b_dec – $a_dec;
}
$mtime = microtime();
$mtime = explode(” “,$mtime);
$mtime = $mtime[1] + $mtime[0];
$end_time = $mtime;
$total_time = ($end_time – $start_time);
$total_time = substr($total_time,0,5);
print”Processing Time: $total_time seconds\n”;
?>

One thing here:

You’ll notice that this uses ROOT priviledges, so I’d suggest you encrypt this script. ioncube does a remarkably good per-script encoding online. Usually, they want (about) .10, which is incredibly reasonable!

SO, now that we have our script, what on earth are we going to do it? Using it is the most important part, right? Let’s get to that:

You’ll need to change the permissions on the script to what you like them to be, but it should (note: shoudl) at minimum be executable

chmod u+x ./scriptname.php

And we’re done

Now, let’s call the script to make sure that it works right

./scriptname.php

If it doesn’t work right, well, there’s usually a problem. Firstly, make sure that php is correct, and in the correct location. From bash:

which php

If it’s not in /usr/local/bin/php (which most are), then adjust the first line of the script
Secondly, maybe you need to update php? That one, I’m not going to cover in a tutorial as it CAN break things. Feel free to Contact Me though for a very well priced update
Thirdly, hey, things happen. If it still doesn’t work, after you’ve updated php AND checked the location, then please, feel free to reply with your problem and we can see if we can’t get it working together.

Now, it works. Let’s tell the system to do it automatically. Again, from bash:

crontab -e

Add the following to your cron job:

30 12 * * * /path/to/scriptname.php > /dev/null 2>&1

THIS will set the cron job to run @ 1230 every day, and then you will have optimized tables.

Thoughts, comments, ideas? As always, put ‘em here.

[root@server ~]# rpm -e httpd httpd-suexec httpd-devel php php-pear php-devel php-imap
service httpd does not support chkconfig
error: %preun(httpd-2.0.52-32.3.ent.centos4.i386) scriptlet failed, exit status 1
[root@server ~]#

What happened here? This was a stock CentOS4 server (completely updated). How to resolve? Read more to find out.

in /etc/init.d/httpd (or /etc/rc.d/init.d/httpd) , on the second line, add

# chkconfig: 2345 10 90
# description: Activates/Deactivates Apache Web Server

This will force httpd to support chkconfig.

Now, you can run the following safely

[root@server ~]# rpm -e httpd httpd-suexec httpd-devel php php-pear php-devel php-imap

Unfortunately, you will need to reinstall your control panel (CPanel, DirectAdmin), as they don’t install correctly when they hit that error. If you don’t know how to do this, that’s quite simple.

Installing Cpanel:

wget http:// layer1.cpanel.net/cpanel-universal-install-v11.sea (take out the space after http://)
sh ./cpanel-universal-install-v11.sea

Then sit back and wait for the stuff to go on. It’ll take 1-2 hours to install CP, then you need to re-install rvskins and Fantastico if you have them
Fantastico

cd /usr/local/cpanel/whostmgr/docroot/cgi
wget -N http:// files.betaservant.com/files/free/fantastico_whm_admin.tgz (again, take out the space after http://)
tar -xzpf fantastico_whm_admin.tgz
rm -rf fantastico_whm_admin.tgz

Rvskin

mkdir /root/rvadmin;
cd /root/rvadmin;
wget http:// download.rvglobalsoft.com/download.php/download/rvskin-auto/saveto/rvauto.tar.bz2; (again, remove the space after http://)
bunzip2 -d rvauto.tar.bz2; tar -xvf rvauto.tar;
perl /root/rvadmin/auto_rvskin.pl

Now you’ve got cpanel re-installed properly. Hope that helps out

Installing Directadmin:

wget http:// www.directadmin.com/setup.sh (again, take out the space after http://)
chmod 755 setup.sh
sh ./setup.sh

Did this help you out? Hey, leave a comment and say so!

The server was configured to not permit you access to the specified resource. If you believe this is in error or inadvertent, please contact the
system administrator and ask them to update the host access files.

There IS only one thing causing this through CPanel, a properly configured server, at least the way I set servers up there is. With a great deal of investigation (ok, it only took me about 3 hours, within the first week of using CP 11;)), I found the issue. The problem? It’s NOT a server issue, but an ISP DNS issue. Here’s what happened, why it’s happening now and not before, and what the ISPs are saying about it:

Firstly, what is happening, and why it’s happening now:
Servers should always be configured to deny host spoofing attacks, through any method available. One simple method? Adding

ALL: PARANOID

to /etc/hosts.deny

What does this do? Well, that’s actually quite simple:
This entry in hosts.deny will deny access to ANY program utilizing tcpwrapper security unless the DNS is properly setup. If it is not setup properly, the activity is denied. Simple, right? Still don’t understand? Ok, I’ll go a bit more in depth here:
Let’s take an ip I own, lease, etc, and use that as an example. This ip is 65.254.41.186
Now, doing a lookup on 65.254.41.186 shows the following:

nslookup 65.254.41.186
Server: 208.67.222.222
Address: 208.67.222.222#53
Non-authoritative answer:
186.41.254.65.in-addr.arpa name = jarlata.linux-tech.net.
Authoritative answers can be found from:

This is good. I have RDNS setup for this ip address to point to the hostname of jarlata.linux-tech.net . Now, according to RFC standards, I must ALSO have a SINGLE A entry in my dns configuration for jarlata.linux-tech.net, and it MUST point to the ip address 65.254.41.186. I do, and it does

nslookup jarlata
Server: 208.67.222.222
Address: 208.67.222.222#53

Non-authoritative answer:
Name: jarlata.linux-tech.net
Address: 65.254.41.186

So, by doing this, I’ve passed a basic security check, and first year networking classes.
The problem, however, comes when ips are NOT properly configured. For example, let’s take a very odd situation, let’s say I had 127.0.0.1 configured to point to unlocalhost as a reverse dns entry. Of course, we all know (those of us in the security game) that’s not right, and it’s wrong, but let’s say for the sake of argument that I did. If I did NOT have an unlocalhost entry in my FORWARD dns (the zone responsible for 127.0.0.1), then , well, I wouldn’t be doing things properly, now, would I? Of course not. Even better, if I had unlocalhost pointing to 127.0.0.1 and 127.0.0.2, it would be, again, an incorrect entry.

Now, what all affect does this have on servers, security, and the like? Believe it or not, a LOT!
In the first case (my host), the DNS is configured properly, so traffic walks in through tcpwrappers, no worries , walks out, and things are normal. They’re normal because they were setup properly!
In the second case (my unlocalhost), things are NOT configured properly, so traffic walks up to the tcpwrapper security guard, and gets booted out. WHY? Because it’s not possible to verify the authority of the host requesting traffic. What? Why SHOULD anyone leave a server open to traffic which can not be tracked back if need be? You SHOULDN’T!

The problem here is that without this check, it’s entirely possible to setup a fake sub.domain.com check and get into a server. I’ve seen it done enough times (not on my servers mind you) to know it IS entirely possible! For example, I could setup a server and ip with the hostname of suck.microsoft.com , ssh into a server, do my damage and leave. They, in turn would see that I came from suck.microsoft.com and think “hrrm, that’s odd”. Of course, they’d have NO way to know where the original attack came from, and NO way to pursue any sort of investigation.

So, that said, on to the LAST two questions:
Why is this just now taking effect on Cpanel related services? Truth be told, CPanel is just now getting in the game of increasing security and awareness of it., and they’re starting to be mroe and more open to using tcpwrappers which have been built into the Linux environment for ages now!

How to get this resolved?
That one is tricky. In order to get this issue resolved you will need to have your ISP setup PROPER dns for your ip and all of their ip addresses. If your ip is 1.2.3.4 and has an rdns entry of ip-4-3-2-1.provider.com , then there MUST be an Authoritative entry and ONLY one authoritative entry for ip-4-3-2-1.provider.com pointing to 1.2.3.4 . No other A entries may exist for ip-4-3-2-1.provider.com .

Your basic ISP tech will tell you that they can’t help you, because, guess what, they can’t. In fact they’ll try to tell you that you need to have a tech come out. Well, guess what, THAT isn’t going to solve anything either. This isn’t something that can be solved by some Joe Bloe in a cable (or ISP) truck (or van), it must be solved within their level 2 (or 3) network administration setup. You’ve got it, you need to talk to the bigwigs, and you probably won’t have much luck with them. Have them add an A entry for your IP, or FIX the A entry so that there’s only ONE in there.

In the past couple of months, I’ve had the unfortunate experience of dealing with ISP’s left and right about this issue. While MANY of them are co-operative, many just choose to be ignorant about proper RFCs, standards, and refuse to change their dns so that it is PROPERLY adherent to standards. Much like Microsoft trying to use IE to write their own standards, ISP’s are doing this left and right. More notably, the ISP’s who have repeatedly REFUSED to do things and correct their mistakes?

AT&T
ComCast
Hargray (1)

1 -

One of our support reps here at Hargray Communications forwarded my
group your e-mail. I help maintain our DNS servers, and hopefully I can
shed some light on this situation.You’re right, we don’t have A records for the names we have in our PTR
records. We consider this unnecessary, and would just add to clutter in
our nameservers’ configurations. The only RFC I can find that says this
is required is RFC 1912, which is not an “Internet Standard” RFC. If
you can point to an Internet Standard document which says this is
required, we would appreciate the opportunity to peruse it.

Yeah, it’s unnecessary to provide authentication of an IP, and password auth really isn’t necessary either. ALL authentication methods and security methods are valuable. An ISP claiming “it’s not necessary to provide A entries for DNS” is just wrong!! Clutter? Maybe, but if you don’t CHANGE it, and keep it all in one zone, that takes one guy an hour (MAYBE an hour) to edit a zone file for all ip address. Hire a flunky to do it! it’s simple text editing!

As of Cpanel 11, Apache 2 is now the default installation. The apache build script is completely rewritten (or so it would appear), the way it works is quite new, at least as it results to Apache 2.

Seeing that they had declared it the default, I figured I’d give it a go, try to figure out what could about it and to get things working in a stable, production environment. After all, why not , it is after all what individuals expect of me, right? To try to figure out how to make things work? Well, it took about 8 hours, but I do believe I’ve finally got ‘er figured out and ready to roll.

So, what was/is the problem?

Originally, the CPanel build failed to build http at all. Not a real issue, I’ve dealt with that before. It took about an hour to get a stable build of apache2 on one server.
Once CPanel had built apache2, I had to recompile php. With the custom configuration I use (patches/updates/etc), I had to completely rebuild it from the ground up. Of course, this meant rewriting my own build script (soon to be released, more info on that later), but that was due.
Once my own build script got rebuilt (another hour), and php was recompiled, I noticed that ioncube wasn’t loading right. My thoughts were :
WTH, it’s loading fine in the CLI (shell)?
So, now, 2 hours after the rebuild process, I have to try to revert things. This time, not only on ONE server but both, because I had actually thought (foolishly) that the problems were resolved. Silly, silly me!

Well, reverting things didn’t even come CLOSE to going as well as it should have, and, yeah, I should have expected as much (right?). I mean, regression NEVER works as well as you want it to. So, after 2 failed builds, I just kept digging in until the problem was found.

The problem? Apparently Zend Optimizer wasn’t loading. Since the ion loaders are built off of Zend (even standalone), thusly, ion couldn’t load. Not a huge surprise there. So, how did I solve the issue? Well, with a little help from the ion faq, I was able to find out that –enable-versioning needed to be turned OFF. From their website:

This option can prevent the export of global PHP API symbols, causing failure when attempting to link libraries such as the Loader or Zend Optimiser. PHP must be rebuilt without that option so that the PHP API is correctly visible. A phpinfo page should show at the top of the page what options were used to configure PHP, and should confirm that the option had been used.

So, after about 8 hours of solid hair pulling, investigation, and the like, FINALLY I have a working php/apache2 build on both servers. Talk about a frustrating experiment gone wild, and all starting out when I just wanted to update php. FUUUUN!