In preparation for a switch to a new invoicing system, our new bookkeeper recently closed a lot of old PayPal invoices, many of which were paid by check (and left open on PayPal). Inadvertantly, the automated “Invoice Cancelled” notice was allowed to go out - we usually stop those from being mailed because they are confusing. We apologize for the confusion.

Please consider it a sign of progress on our new support and billing system. Invoices that come to you in the future should be easier to understand and less PayPal-oriented. August will be a transition month, and some new billing policies will be announced for September.

Some of you may have noticed our Gryphyn Media site was down for most of yesterday. We were hacked - it was the “iFrame exploit” that is ripping through a lot of shared hosting environments. It compromised this WordPress installation. We committed the cardinal sin of letting it become outdated, and a new exploit took advantage of that.

We cleaned it up and were going to upgrade WordPress, but our datacenter offered to upgrade our VPS server with a “more secure OS.” At the time, the wisdom was that our Fedora Core 2 OS was more vulnerable - that has not proved not to be the case. But early in the attack, the pattern made it look like FC2 was the vulnerability, so we agreed to move to a Centos. In retrospect, we wish we hadn’t. The upgrade itself went smoothly - nothing wrong with Centos - but CPanel also upgraded, and that’s where it got ugly. The newer Cpanel versions have a different mail folder structure, and the backup would not restore properly. We had to manually rebuild the DNS zone, mail directory structure, and reinstall software and databases.

We thought we would be down for half an hour at 4 AM - it was 16 hours. Oh, and Comcast had network problems in the middle of it. Both Tracy and Warren use Comcast for an ISP - their connection speed was like dial-up for several hours. They had to phone the client datacenter in VA to check the client servers.

Fortunately, we have DNS redundancy, so our DNS zone stayed available. We also have backup email queuing, so emails that were sent to us during that time were stored on another server and delivered when email came back up.

But we were essentially “out of touch” yesterday, and we are sure that alarmed some of you. We apologize. Had we suspected the upgrade would be so painful, we would have handled it differently.

No customer sites or email were affected. No client passwords were exposed. The exploit did not involve our desktops or laptops, and client password and contact data is managed with a secure application. Our company sites are in a completely different datacenter, so that we stay up if something catastrophic affects client servers. And if something horrible happens to us - you stay up.

We are wading through saved email this morning and will contact those of you that emailed. We will also be contacting some of you about your own exploitable PHP scripts that have become out-dated. Don’t let this happen to you! The iFrame exploit is causing lots of trouble for many hosts and their clients.

(At least our bad day was not as bad at yesterday at the 365 Main datacenter in San Francisco, where major sites, like Craigslist and the whole Six Apart blog network, were taken down by a power outage. An angry mob gathered.)

We are experiencing a network issue this evening, causing some email to gryphynmedia.com address to be rejected with a “550 Error.” This affects only gryphynmedia.com addresses, such as support@gryphynmedia.com. Please use the helpdesk to contact us while the problem is resolved.

Update: The problem was found and fixed - not network; it was a configuration problem. A cpanel update messed up the DNS recursion, which breaks email, since the server can’t look up any hostnames. We apologize for any trouble contacting us. Email is now delivering normally.

Email and server speed are being affected on Champion by a script sending botnet phishing emails. Over 15K messages went out in the past hour, mostly to invalid hotmail addresses, creating a huge number of bounces and tying up Exim. Several people are working on identifying the PHP script - not always easy. We expect normal server speed to be restored shortly.

Update: 8:55pm - Server speed is improved and the phishing process has been killed. We are still investigated the source of the script.

A website on Champion is being attacked with large amounts of web traffic and email. It is not clear why anyone would attack on an otherwise ordinary real estate site. We are blocking what we can, but the attack was causing some connection trouble earlier this afternoon. It appears to be dying off. We are watching, and consulting with a security expert.

Last night’s cpanel update on Crown apparently caused SSL to freeze on Crown. The problem didn’t rigger any monitors on our end, so unfortuantely we did not know about it until tickets came in. Restarting Apache appears to have resolved the problem.