Cake PHP Set Class - overlay function

The CakePHP Set Class is a wonderful collection of functions that allow you to handle and process CakePHP arrays quickly and efficiently. It has features that deal with stripping out data, changing arrays to objects, and custom merge functionality.

Recently however I had a problem that was bothering me, and I thought some sort of set functionality would do the job. (Please tell me if this functionality has already been written).

Here's the problem:

I have a table that I download periodically and I want to update it into my database. The problem is that I have custom ID's for each row of my current table that are indexed throughout the rest of the database. I've found that I need to update existing records, delete any records that are not in my downloaded table, and I need to add new records that don't exist in my current table.

Here's an example:

On a regular basis, you get a list of 5000 products that your online store promotes from your affiliate. Once a week you need to download all of the products and update your products table. A standard drop and re-insert will not work, because you use custom id's in your products table that are referenced in your customers_products table and others. How do you update these 5000 records?

The answer is to think of it in terms of sets...

In the diagram, Set A is my current table that I want to update. Set B is the new table. Everything that is in Set A and not in Set B, I want to delete (Yellow). Everything in Set B not in Set A, I want to insert (Dark Green). And Everything that exists in both I want to update (Light Green).

Breaking the problem into a sets will allow you to realize what you need to do. One way of looking at a possible solution is to find what you need. From the diagram, all you need is everything in green (light or dark). In our product example all you need is a set of new products while maintaining the light green Primary Keys. (the dark green will generate new id's).

Implementation:

function updateProducts() {
$products= $this->Product->query("select distinct(prodid) as `match_id` from import_products as `Product`");
// Make the products array look like cake arrays...
foreach($products as $i => $product) {
foreach($product as $model=>$this_data) {
unset($products[$i][$model]);
$products[$i]['Product'] = $this_data;
}
}
$current_products = $this->Product->find('all',array('recursive'=>-1));
$res = $this->overlay($current_products, $products, "{n}.product.match_id");
$this->Product->query("truncate table products");
$this->Product->saveAll($res);
}

// function overlay:
// This function returns the $to array with merged $from elements.
// $from - array[0..x] of cake based array structure
// $to - array[0..x] of cake based array structure
// Path field for the overlay using set::extract's strucure
// eg. $path = "{n}.TableName.fieldname"
// * the Path must exist in each element in the $to and $from array
function overlay($from, $to, $path) {
$from_ary = Set::extract($from, $path);
$to_ary = Set::extract($to, $path);
if (count($to_ary) != count($to)) return $to;
if (count($from_ary) != count($from)) return $to;
$flipped_from = array_flip($from_ary);
$cnt = 0;
$results = array();
foreach($to as $key => $val) {
if (!empty($flipped_from[$to_ary[$cnt]])) {
$fkey = $flipped_from[$to_ary[$cnt]];
$results[$cnt] = Set::merge($from[$fkey], $to[$cnt]);
} else {
$results[$cnt] = $to[$cnt];
}
$cnt++;
}
return $results;
}

What does all this do?

Put simply, I gather my SET B (the new updated products that I downloaded), and then my SET B (my current set of products). I then call my function overlay(). Overlay simply returns SET B with elements from SET A merged with it based on the Path field. What is nice about this concept is that it allows you to simply pass the results overlay() directly to a cake saveAll() function.

I would not suggest using this code for giant tables. I did it on a zip code table with 42000 records, and the saveAll took a few minutes to rebuild the table. This may not be the most efficient solution, but it's clean and useful for "once-in-a-while" updates on tables.

How to Add CAPTCHA Security Images to Wildflower Comments

So you love Wildflower CMS for CakePHP, and you've used it to create a number of blogs. However you've noticed something. You are getting comment spammed.

Put simply, Wildflower doesn't have a good solution for preventing comment spam.

Initial Setup

1) Download the latest Wildflower development copy to your system. You can find the latest code for Wildflower on GitHub.

2) Set up your development environment. This includes:

• Apache virtual host setup to point to your wildflower/app/webroot folder
• Setup your local database called "wildflower"
• Modify your hosts file so that it points to the Apache domain
• Restart Apache.

3) At this point you should have a working copy of Wildflower.

Download Securimage CAPTCHA

You can download Securimage here:

Securimage Download

• Grab the zip file and unpack it into your app/vendors/ directory.
• Download the words.zip file and extract to app/vendors/securimage/words/
  

Building the Dynamic Image

• Download the Captcha.php script and place it in app/webroot/

• Edit app/webroot/.htaccess and add the following line:

RewriteEngine On
RewriteRule ^img/captcha.jpg$ captcha.php [QSA,L]
RewriteCond %{REQUEST_FILENAME} !-d
...

• Test the image by going to http://yourvirtualhost/img/captcha.jpg

Create the Form View

Copy

• ~/wildflower/views/wild_posts/view.ctp

to

• /app/views/plugins/wildflower/wild_psts/view.ctp

(Create the directories as needed)

• Now edit /app/views/plugins/wildflower/wild_psts/view.ctp and add the following two lines in the Comments Form.

$form->input('url', array('label' => 'Website URL (optional)')),
$form->input('content', array('label' => 'Message', 'type' => 'textbox')),
$html->image('captcha.jpg', array('alt'=>'Code', 'id'=>'security_image')),
$form->input('security_code', array('label' => 'Enter the word.')), $form->hidden('post_id', array('value' => $post['WildPost']['id'])),


Process the Form

• Open up ~/wildflower/models/wild_comment.php

• Add the following Validation Rule:

public $validate = array(
'name' => VALID_NOT_EMPTY,
'email' => array('rule' => 'email', 'message' => 'Please enter a valid email address'),
'url' => array('rule' => 'url', 'message' => 'Please enter a valid URL', 'allowEmpty' => true),
'security_code' => array('rule'=>'validSecurityCode', 'message'=>'The word you entered was not correct.', 'allowEmpty' => false),
'content' => VALID_NOT_EMPTY
);

• Add the following Function:

function validSecurityCode() {
if (!empty($this->data[$this->name]['security_code']) && strtolower($this->data[$this->name]['security_code']) != strtolower($_SESSION['securimage_code_value'])) {
unset($this->data[$this->name]['security_code']);
return false;
}
return true;
}

Try to add a comment to a blog post and see what happens.

Writing Bad Test Cases

So, you spend days and days writing test cases and you find out that they become more of a hassle than you expected. Here are some common mistakes in writing test cases which may help you in your test case development.

1) Your test case creates live data. When you do a quick test case to try to make sure your function works, sometimes you simply set it up to create a new record in your database. This can happen in several ways.

• You connect to a Live API with your test case. This could actually send credit card transactions, or send live records to your other application, which dilutes live data with fake records. This could lead to bad reporting in the long run.
• You connect to your live database to test your functions. This is a really bad idea. Make sure your connection string points to a test database or you may end up losing mission-critical data.

2) You change your code to accommodate for test cases. This is a mistake as well. Your code should do what it was intended to do. If you need to add debug lines or incorporate new functionality into your code so that test cases will work, then chances are you either need to go back and study up on test case development or you need to redesign your code so that test cases will not interfere with the system process.

3) You make test cases depend on each other. This is a temptation that should be avoided. When you are running test cases, each function should be independent, even though test cases run linearly. You should not have one test function that presets data, and then assume that that function was called by the test suite.

For instance, let's say I have a test called testLoadUserData and that function loads user information into the variable $this->userdata for testing. Then under than I have a test for testShowUserData. It would be a mistake to assume that $this->userdata still exists in testShowUserData. Why? Because sometimes we comment out tests so the page doesn't take so long to load. This could breat testShowUserData if it depends on other tests. A better solution would be to write a loadVariables function and call it in each test case.

The goal of test cases is to help write modular functions. Often times poorly designed code will cause problems in developing test cases. If you write bad test cases, just consider it a learning experience, and keep at it until it works well.

Good code is poetic in nature.

5 Tips for Cake PHP Development

1. Don't over-complicate your code.
   
   
2. Work within the constraints of Cake.
   
   
3. Make it work, then go back and make it work better.
   
   
4. Skinny Controllers, fat models.
   
   
5. Test Test Unit Test.
   

The Ultimate Windows PHP Development Environment

So, you are a PHP developer and you have windows. You've tried all kinds of different editors and setups on your system, but nothing works the way you want it to. Well, your answer is here. I've compiled a list of applications that are essential for PHP development.

Hardware

A Computer - Developers do not need fancy machines to develop. They just need something within the past few years that has enough ram to run modern browsers and a bunch of apps open at the same time. I would suggest a minimum Windows XP. Vista will do as well.

2 Monitors - While a single monitor is nice, having a large area to work is also very helpful. What I suggest is adding a video card that supports two monitors. Then plug the monitors in side by side, and configure them.

The reason I suggest 2 monitors rather than having a wide-screen display is because I would suggest that you virtually stack your monitors so that one screen is virtually "above" another.

Why above one another? Simple. Programmers are lazy! 1) By having your monitors placed above one another you simply need to slide the mouse up to get to the second screen. The advantage of this is that the mouse never needs to leave the mouse pad. 2) Having the monitors setup side-by-side makes closing programs more difficult. If I have my browser maximized on screen 1 and I use the mouse to close it, I move the mouse to the right wall, go up to the x and click it.


Basic Software Tools

Browser(s) - Of course you need a browser or two. Being a PHP programmer doesn't always require that you have awesome xhtml/css design skills, but it's worth at least having several different browsers available to test your system on. There are a number of options available, however a simple set of browsers would be most useful.

• Firefox - This is a Must! If you don't use firefox the only excuse you have is to be using Google Chrome. Other than that, there is No excuse. Firefox has a few nice extensions that make web programming helpful. Often times we will need to integrate PHP back-end with Ajax, or something. For testing purposes in this situation, it would be helpful to add Firefox Firebug.
• Google Chrome - As mentioned earlier, Google Chrome is a good alternate to firefox. It is still relitavely new, but has some good potential.
• Internet Explorer - Ok, yes you need it. Not for browsing, but for testing. It is useless to have a site that looks great on Firefox when 90% of your users are using Internet Explorer. In addition to this it is important that you have multiple versions of IE. I would highly suggest: Multiple IE.

PHP Editors - I'm certain this may be one of the more contriversial sections of this post because everyone seems to have their own preferences for editors. However since I'm posting this, my editor picks are best. :-p You need different editors for different reasons. There are two primary editors I use for different reasons.

1. Notepad++ - This is by far the best editor I've used in a long time, and it is my favorite. Why you may ask? I judge editors by a number of factors. 1) How little memory they use 2) Easy to use Tabs 3) Highlighting 4) Useful features. Notepadd++ has ranked #1 for all of these factors in my book. Here are a few things I like about it.
   
   - The Light Explorer plugin makes browsing my directories quick and easy.
   - It comes with the ability to edit files directly on a web server through the ftp_syncronize plugin
   - It opens very fast, takes very little memory, has nice highlighting, and tabs at the top.
   - It opens and closes function tabs letting me see only the code I'm working on at the time.
   
   
2. gvim for Windows - Before coming across Notepad++ I used gvim for windows exclusively. Why? Having a background of using the command line, I learned Vi. When you ssh over to a server and need to edit a file, Vi is by far the best editor (sorry emacs lovers). Well, since I'm always using :wq or %s/.../../g and other vi commands, I found that using VI in a windows environment was great. It allowed me to quickly edit files and use my favorite vi commands.
   
   Why did I switch? Mainly because of tabs. While I know the basics of VI, I never could setup a good solution for adding and managing tabs. I know there is a VI solution, but I fell to the temptation of opening 10+ Vi windows while working on a project. The transition to Notepad++ was a careful one, but I still keep Vi around.
   
   Why should you consider keeping Vi around?
   - It again is light weight and fast.
   - It offers nice highlighting.
   - It's VI! So you have the great features of VI.
   
   What can you use it for?
   - Various projects with CSV Files or Excel files. For instance you can turn an excel sheet into a csv file by copying the data to VI and running the following command:
   
   %s/\t/,/g
   
   - Vim is great and fast for formatting data quickly and on the fly.
   - The latest version of VI can open very large files quickly.
   
   Oh, and don't forget, it's VI!
   

Other Editors - for those of you who are just getting into PHP or if you need help for function names, or if you want to pay for your editor, I suppose you could use apps like Dreamweaver or SciTE or Eclipse with PHP.

The Applications

PHP, MySQL, Apache - This is commonly known (on windows) as WAMP installation. You will need three basic applications for 90% of your PHP development.

1. PHP - Of course you will need PHP. This is the application that you actually use to run the code you're developing. I would suggest keeping a browser open and making use of the PHP function search box on the top of the php site. I would not suggest installing php alone on your machine unless you have the time to configure it yourself. (see below for xampp installation)
2. Apache - Apache is the most popular and most stable webserver out on the market. It is worth learning, because if you go to work for a new company, chances are that their web servers use apache. (again, see below for installation)
3. MySQL - There are a number of databases out there, but by far, mysql is the best for mid or even high traffic sites. For many years it has been second to Oracle, but recent versions have made it highly competitive to Oracle. And of course, you can't beat the price

So what is the best way to get these apps onto my windows machine? I've tried a number of applications that try to tie php, apache, and mysql into a windows environment where it is seamless and allows you to not have to install individual elements. By far, I have found that Xampp is the best of the best.

Xampp is an easy installation process that installs a fully working and functional WAMP server onto your system. It comes with a handy little control panel to allow you to stop and restart the apache and mysql processes. Not only that, it comes with PHPMyAdmin pre-installed to allow you to quickly get your database up and running. Note: Besides the default install, I would suggest making one additional modification to your xampp install. Open C:\xampp\apache\conf\httpd.conf in your favorite editor (Notepadd++ of course) and uncomment the line that starts with "LoadModule rewrite_module"... around 118 or so. This will turn on Mod Rewrite to allow a number of applications work with/nice/urls.

Frameworks

CakePHP - A while back I spent some time looking into frameworks. Out of all the frameworks I came across, CakePHP stood out above all the rest as a fast, reliable, well-designed framework. After spending a year and a half writing code in CakePHP, I've found that it is one of the most flexible frameworks I've used. Since then I've released over 20 sites using CakePHP that get hundreds of thousands of unique visitors per month. It has bee completely stable and reliable. It also has a growing community of developers who are constantly coming up with different uses, plugins, and add-ons.

Forgive me for not adding other frameworks, but feel free to explore your own.

Useful Utilities

VirtuaWin - This is a Must have. I keep a copy of this program on a zip drive in my wallet and install it on any computer that I'm using. Basically VirtuaWin allows you to have up to 20 desktops on your computer. You setup your shortcut key () to navigate to the desktop you are working. This allows me to work on several projects at the same time along with having one reserved for email and music controls. Once you use this, you'll never go back to a single desktop. Plus if you have dual monitors, you double your desktop capacity up to 40! Yikes!

CoreFTPLite - Of course you need an extra FTP utility. I found that Core FTP Lite does everything I want it to do. Fast loading, fast transfers, SFTP, doesn't crash, easy to use, etc. What else can you say? Filezilla is nice too.

Putty - Never go anywhere without putty. Putty is a small executable terminal window that allows you to ssh anywhere on the web. You don't install it, you just open it. I usually copy it to program files / putty and create shortcuts to it for consistancy.

RealVNC - If you have multiple windows computers and again you are too lazy to get up and log into that computer, VNC is the way to go. Vnc lets you open the desktop of another machine on your computer. It's a fairly simple setup, and works great.

Windows XP Tricks to save time - This is a video that suggests additional cool features to help you save time when programming on windows. I use a few of these techniques as well.

And Finally... The best for last...

Git

Git - I put git in its own category because it's so awesome and it is a must-have for ALL development. Git is a version control repository, but far better and faster than CVS or SVN. Git allows you to store snapshots of the files you are working on and easily recover bad changes. Take a look at one of my previous post about using Git with SVN for help in this area.


The End.

Unit Testing Cookies in Cake PHP

One problem I came across when building unit tests was trying to check to see if my cookies were setup properly. (Yes I still use cookies.) The problem with trying to unit test a cookie in Cake PHP is that you are only allowed a single page load for the test. A cookie doesn't exist until it is sent to the browser and the browser sends the cookie back with a new page load.

My first thought was to place a conditional in the test case. This would basically be something like:

if (!empty($_COOKIE['id'])) {
$this->assertTrue($_COOKIE['id'] == 100);
}

This method would require you to have to reload the page to properly test cookies. The first page load will consist of 0 tests. The next page load will consist of 1 test. But this method didn't seem like the best way to go about it.

Then I discovered the PHP function: headers_list().

Headers List will return all the headers that you are about to send to the browser in an array format. This was the solution!

Now I can test what the function is expected to send to the browser (which is independent of the browser actually accepting the cookie or not). Here's a sample of what I did.

function testStoreCookie() {
$this->Controller->storeCookie(100);
$header_list = headers_list();
$cookie_true = false;
foreach($header_list as $item) {
$cookie_true = is_string(stristr($item, "Set-Cookie: id=100")) || $cookie_true;
}
$this->assertTrue($cookie_true);
}

There you have it! You can now test setting cookies without reloading the page.

Cheat Sheet Collection

Programming Cheat Sheets

Have you ever been programming and simply forgotten a command? When ever that has happened to me, I've had to go to google or the api website to find documentation, and many times that takes longer than I want. Then I discovered programming cheat sheets. Programming cheat sheets are ethical, time-saving, tree killing documents that have commonly used functions, procedures, commands, or examples to jog your memory when programming.

For your convenience, I've put together a zip file with 13 Cheat Sheets. Below is a description of some of the ones you will find in this collection.

You may download it here.

---

CakePHP Cheat Sheet

The CakePHP Cheat Sheet is helpful for all of your baking needs. It includes naming conventions, common model commands, controller commands, view commands, helper properties, globals, and component information.

Cake can be found at http://cakephp.org

---

HTML Character Sets

The HTML Character Set Cheat Sheet is quite useful for finding out what the & character is or for escaping "'s. Also it is useful when using template engines that have problems with characters like { or $.

---

HTML Color Chart Cheat Sheets

This is an essential for designers or when you are coding CSS on-the-fly. It is also useful when picking colors for a new website.

---

CSS Cheat Sheet

Most web designers have their favorite CSS documentation site bookmarked. Many use reference books to lookup css attributes. This cheat sheet is useful for both novice and experienced CSS'ers to serve as a reminder of the names of css attributes.

---

Git Cheat Sheet

Git is a great repository system that allows you to archive all of your work quickly, and be able to keep your large project organized. Similar to SVN or CVS, Git offers a local repository as well as public repositories for sharing code. The cheat sheet is very valuable as a reminder of what commands to use for using Git.

---

HTML Cheat Sheet

Are you often forgetting what HTML is able to do? This cheat sheet has a list of much of the markup that you may not normally use, as well as those old favorites.

---

Also included are:

• Javascript Cheat Sheet
• Jquery Cheat Sheet
• Mysql Cheat Sheet
• Php Cheat Sheet
• Prototype Cheat Sheet
• Regular Expressions Cheat Sheet

The full download is available here.

Feel free to post additional cheat sheets in the comments section.

Four Ways To Retrieve Model Data

There are a number of ways (in addition to direct model access from a controller) to retrieve data from a model. The type of method you use depends on what you are attempting to do. Here is what I've discovered.

$uses array

As you recall, the standard way a controller defines its models is by using the variable $uses.

$uses = array('Post','Comment');

This works fine for most cases, but what happens if one action in your controller needs to use the UserNotes Model, but no other actions in the controller needs it. One way to solve this would be to simply throw it into the $uses array().

$uses = array('Post','Comment','UserNote');

Now in your controller action you can call $this->UserNote->find... to get the data you want to get.


ClassRegistry::init

Recently, after browsing through the regular CakePHP blogs, I came across an article about building a dashboard in CakePHP. This article introduced the available function called ClassRegistry::init(). I found it to be an interesting addition to my growing Cake PHP toolset. ClassRegistry::init() is the actual method that a controller uses to load its models (you can find the call in the cake controller class in the function loadModel).

When to use $uses and when to use ClassRegistry::init

There may be times when you do not want to use the $uses array. In the above example, UserNotes will be loaded for all of the actions (even the ones that never call it). This is a problem because the more models that that single action needs, the more strain it will put on all the other actions. The question of when to use ClassRegistry::init is: Are you willing to have all of the other actions suffer from the performance hit of loading unused models?

This is where ClassRegistry::init() comes in to play. ClassRegistry::init() will load a model directly, skipping the standard model caching, from within the action. This means only the action you are running at that moment will have UserNotes loaded. All of the other actions in that controller will not load that model.

function getUserComments() {
...
$userNote = ClassRegistry::init('UserNote');
$userNotes = $userNote->getUserNotes($user_id);
...
}

When should you use ClassRegistry::init?

1. When you have an action that calls models that other actions do not need.
2. When you need to retrieve unrelated model data.
3. When you want to skip model Caching and access the model directly.
   

Model Relationships

Many times both of the previous methods can be avoided because of natural model linking. Remember, when you define a model, you also setup relationships. The controller/action can use those relationships to retrieve data, as seen in Mark Story's Blog post.

If you are requesting information from a joined table within the relational chain, calling models through their relationships would be the best methodology.

$uses = array('User');
...
$this->User->Notes->find('all');

Although this method is limited, it is powerful, and it prevents redundency. By default a model will load the models it is associated with. So when you want data from an associated table, don't bother defining it in the $uses array, because Cake will automagically load that model and make it available through your current model. The key to making this work effectively is good database design.

requestAction() a Last Resort

The tackiest way to retrieve model data is to use requestAction. The reason for this is because of how Cake works. As you can see from the diagram, the dispatcher determines the route and then loads a single controller along with its models. Using a request action will cause the dispatcher to run again (possibly re-running routes) and then loading your additional controller, along with its models. This can cause a performance hit without caching.

Inside your Users controller you have an action called show_notes(). show_notes needs to access the notes table and retrieve a list of notes.

// Users Controller
function show_notes() {
...
$notes = $this->requestAction('/notes/get_notes/'.$user_id);

}
// Notes Controller
function get_notes($user_id) {
return $this->findByUserId($user_id);
}

There are a number of problems with this type of design.

1. Additional server load for every request, which may cause bottlenecks.
2. It goes against the "fat model, skinny controller" concept
3. You are redefining the controller to do what the models are supposed to do.

Summary

Put simply, there are plenty of options to retrieve model data. The method that you use truly depends on your specific problem that you are addressing.

Using Git with SVN

Although git might generally be recognized as Southern slang, in web developing, "git" is one of the most useful applications for your local development environment. Git is a version control system similar to SVN or CVS, however using git is a bit different SVN or CVS. In this post, we will assume that you are familiar with version control systems, and you are currently using or have used SVN.


Learning Git

There is usually a little learning curve when looking into a new concept. Git is no different, however it is well worth the time investment to learn, because it will certainly save you time in the future. Here are the pro's and cons of learning git:

PRO's

• Increase productivity and speed in local developing - Git is fast and it works locally. Git allows you to take an entire project, copy it, totally trash it with debug code so you can get the task done, and then completely restore the original files almost instantly.
  
• Less hassle - Git's commands are purposefully easy. With one command you can commit all of your changes. With another single command you can create a new branch and immediately start working.
  
• Keeps you on track and focused - with the branching functionality of Git, you can work on two or three projects in the same code base at the same time. Switching between your code is almost as easy as Alt-Tab in windows.
  
• Git is powerful - Git's merging functionality is very quick, simple, and powerful. There are conflict resolution tools that help you to merge safely without breaking things. It also allows you to go back and fix mistakes and recover lost changes.
  
• Git is local - The entire repository for Git is contained in a .git directory in your root path. This means if you have a project with several hundred directories, you won't have hundreds of .svn or CVS directories. Just one .git directory.
  

CONS

• About the only con of Git is the intimidation factor. Currently there is no TortoiseGit to make the familiar transition from TortoiseSVN. The best way to use git on windows is to use the command line using MsysGit.
  

Where To Start

Screencasts and tutorials are some of the best places to start to familiarize yourself with something new. Here is what I did to learn git:

• Watch some screencasts - I started out with the free screencast from debuggable. It gives you a step by step process on how to setup Git along with examples of building a repository.
• Read the Documentation - Yeah I know, writing code is much more fun than reading about it, but hey, it's worth it. I would suggest reading and focusing on the first 3 chapters of the Git User's Manual. It will provide you with specific walkthroughs and examples of each concept.
• Do it yourself - Now you should be ready to start implementing git in your own projects.
  

Development Process

Here is a development process to get you started with git with a project that you are already working on.

1. Open the git bash window.
2. Navigate to the root directory of your project.
   
   cd /c/www/myproject
   
   
3. Set up your git repository in that folder.
   
   git init
   
   
4. Set up any .gitignore files that you may want. Often this would be for tmp folders, dynamically generated files, or any .svn or CVS folders.
   
5. Add and commit everything
   
   git add .
   git commit .
   or
   git commit -a .
   
   
6. Create a new development branch and switch to it.
   
   git checkout master
   git branch t.setup_test_cases
   git checkout t.setup_test_cases
   or
   git checkout -b t.setup_test_cases master
   
   
7. Now you are safe to start coding your task. You are now working on a copy of your files, everything is backed up and recoverable. Add, Delete and Commit as you go, and everything will be stored to the branch.
   
   7a. At any time during development you can repeat step 6 to create an additional branch to get a high priority item finished, then go back to your main project. You can also have two or three simaltanious projects open at once.
   
   
8. When you're done with your changes to a branch, simply merge back to the master.
   
   git checkout master
   git merge t.setup_test_cases
   #Resolve any conflicts here
   git status
   
   
9. Use gitk to examine your changes to see what happened.
   

And it's that simple.

Using Git with SVN

Now that you know the basics of git, you can implement it alone, or within your SVN repository. Git only adds a single directory .git within the base of your application and it does not interfere with SVN. From the Git side, you will need to filter out .svn directories. The implementation is easy enough. Simply add the following line to the file: ".gitignore"

.svn

Basically the .gitignore file will allow git to ignore any matching file or directory. You can use wildcards to find more specific file matching.

At this point you will be able to use the git steps above within an SVN repository to keep up with your local changes without having to commit to the svn repository as much.

Why Use Git in SVN?

Using git inside of SVN has its advantages and disadvantages. One advantage is you gain the flexability of simple branching without having to download the entire repository from the SVN server. It is like adding tabs to a web browser. You can multi-task, branch and merge all locally, and then once your modification is tested and ready, simply SVN commit.

Let's look at an example.

I have an SVN repository setup in my development environment called /www/coolwebsite/ I created the git repository as described above and I setup my .gitignore. I have a 3 item project list for changes on the website.

• change the design of the home page
  
• create an email autoresponder
• create a contact us form
  

The first thing I would do is decide on one project to start on and branch off of the git master.

git checkout -b t.home_page_design master

This says that I am going to create a new branch called 't.home_page_design' based on the 'master' copy, and I'm going to switch to that branch.

So 'git branch -l' should look like:

master
* t.home_page_design

Now let's start working on the project. I edit the home page and save it. I edit 2 css files and save them. Then I add 3 new images. I do a 'git status' and see:

$ git status
# On branch master
# Changed but not updated:
# (use "git add ..." to update what will be committed)
#
# modified: index.html
#
# Untracked files:
# (use "git add ..." to include in what will be committed)
#
# images/banner.gif
# images/logo.jpg
# images/test.gif
no changes added to commit (use "git add" and/or "git commit -a")

Now simply add and commit to the branch.

git add .
git commit .

If you're using TortoiseSVN you will find that the changes have become red in your file browser. This should reflect the changes that you've done in your files. Before committing to SVN, you first want to merge the changes back to the master branch.

git checkout master
git merge t.home_page_design

When you checked out the master, the red files in TortiseSVN should have turned green, but they should have turned red again when you did the merge. There should be no conflicts here and now the master branch should reflect all fo the changes of your branch. An additonal commit is not necessary.

It is now safe to publish your changes to the SVN Repository.

The next project is to create an email autoresponder. Again we make a new branch.

git checkout -b t.autoresponder master

This time as we start working on the autoresponder branch, our client comes to us and says that they need the contact us form built ASAP, and the other project needs to go on hold. We don't want to lose any of our changes to the autoresponder either. Git handles this quite well.

First commit all of your changes:

git commit -a

Second, create a new branch for the contact us form.

git checkout -b t.contact_us master

Now you can start on the contact us form and still hold on to the changes from the autoresponder. While you are editing and adding files, you will notice your changes reflected in TortoiseSVN. During the middle of a project you can easily switch branches and work on them without affecting any other branches.

You finish the contact us form, commit it to the branch and then merge to the master.

git commit -a
git checkout master
git merge t.contact_us

Now master has the contact us additions. Commit to SVN from the master branch, and then checkout t.autoresponder

git checkout t.autoresponder

Finish the autoresponder and commit the changes. Commit to SVN, and there you have it.

Final Thoughts

As you can see, there are some great advantages to using SVN with Git, and there are a few disadvantages. If you branch for every change, it will force you to keep tasks separate which could be a good thing or a bad one. There is a little bit of overhead during development to keep both repositories in sync, but the flexability that you gain is sure to outweigh any lost time.

Feel free to comment.

Wildflower Robust Templates

I've been toying around with Wildflower templates, and I've come up with a basic structure.

Getting CakePHP Help

So I have a problem and I need to get help. Where do I start?

To a beginner, CakePHP can be overwhelming, especially if you are new to the Model / View / Controller (MVC) methodology. So I've come up with a few tips that may help you get started.

1) Start at the beginning. Cake is big and there are a lot of custom methods that exist. Many times learning something to the scale of cake is intimidating, and rather than jumping in to get your hands dirty, you get confused as to exactly where to start. Well, the best place to start is the beginning.

.htaccess - This is the first place apache will look. Read through the .htaccess files in cake to see how Apache reacts to your url string. In CakePHP, the .htaccess file points to app/webroot folder. The app/webroot/.htaccess file says "If the file doesn't exist, open up index.php. If the file does exist, simply display it."

There you have it. Webroot is your root directory. index.php is the file that is executed for any dynamic content. Open up index.php and you'll realize that it simply runs the dispatcher class.

• Don't be afraid to trace through code.
• Grep (or Windows Grep) is your friend to find what you are looking for.
  
• Don't be afraid to add debug the CAKE source files.

2) Find example code. Most basic types of applications have been built and documented somewhere on the internet. For CakePHP, there are a number of screencasts on cakephp.org. Also check out the Cake Bakery on cakephp.org. Google is your friend. Search for examples, set them up and get them working.

• Invest the time to learn. It will pay off.
• Don't take shortcuts. Take each part and explore it until you understand it.

3) Ask for help. Sometimes you will need help. It may be a specific question about a certain set of code, or it could be a concept question. Asking questions is important, and there are many people willing to answer your question, but remember these important tips for asking questions:

• Make your question make sense. If you can't understand it, the reader won't either.
• Be specific. Give code samples. Give as much detail as you can.
• We don't bite, so don't be afraid to just ask. Sometimes people's responses will help you to narrow down the specifics of your question.
  
• Where can I ask you ask?
• IRC - irc.freenode.net #cakephp (Remember to post your code to http://bin.cakephp.org)
  
• Google Groups - http://groups.google.com/group/cake-php

Other Resources:

Check out these CakePHP Resources: http://cakebaker.42dh.com/cakephp-resources/

Advanced Cake Routing: Dynamic Routes...

I've been working on integrating Wildflower CMS with a number of my sites, however there has been one issue that I've had with how cake routing was setup.

Router::connect('(?!' . $admin . '|' . $prefix . '|login|contact)(.*)', array('controller' => 'wild_pages', 'action' => 'view', 'plugin' => 'wildflower'), array('$2'));

If you notice this route will basically connect everything that is not admin, prefix, login, or contact with the generic Wildflower view controller.

Advantage:

The advantage of having a connect string like the one above is so that we can have dynamic content directly after the URL in the site. Normally the dynamic content would point to something like /pages/name-of-the-article, so the link would look like:

http://localhost/pages/name-of-the-article

However with the route above, everything that is not defined will point to the pages controller, allowing you to skip the /pages/ in the URL. For example, the url above can now be written:

http://localhost/name-of-the-article

This makes it nicer when dealing with dynamic content...

The Problem:

Since Wildflower is a plugin, it is an addition to my normal application. Let's say my normal application has 2 controllers: foo_controller and bar_controller.

Normally in CakePHP, the routes for foo and bar would be automagic. This means that by default if I point my browser to http://localhost/foo/index it will automatically load the controllers/foo_controller.php, and look for the function index() definition. Otherwise it will throw an error.

With the route above, my "foo" controller will not load unless I specifically define a "foo" route to point to the foo controller before the Wildflower route:

Router::connect('/foo/*', array('controller'=>'foo', 'action'=>'index');

Router::connect('(?!' . $admin . '|' . $prefix . '|login|contact)(.*)', array('controller' => 'wild_pages', 'action' => 'view', 'plugin' => 'wildflower'), array('$2'));

Well I don't want to have to define 1 or more routes for every controller I build in my application. I want the automagic to still work.

Solution:

There is a solution for this problem. It is a bit tricky, but Cake PHP is flexable enough to handle it. The solution is to grab only a list of the pages that you want available for the routing, and only assign those routes. This will remove the "catch-all" route from above and allow us to create the normal automagic routes for my application.

But how do you get a list of the pages from the database when the routes have no access to the databse when they're called?

The answer is the object method: requestAction(). requestAction can be called anywhere in the system and it can call any current route. The results of the requestAction call is an array of returned results. Since the router extends object, it has access to call it, and therefore can gain access to dynamic data.

Here's an example:

// Set a temporary route
Router::connect('/pages/get_root_pages', array('controller' => 'wild_pages', 'action' => 'get_root_pages', 'plugin' => 'wildflower'));
// Now request that temporary route.
$root_pages = Router::requestAction('/pages/get_root_pages');
// Loop through the root pages and manually define the pages.
foreach($root_pages as $i => $page) {
Router::connect('/' . $page['WildPage']['slug'], array('controller' => 'wild_pages', 'action' => 'view', 'plugin' => 'wildflower'));
}

The next step is simply to add function get_root_pages() in your pages controller. It would simply find all the active pages with a parent == 0.

The disadvantage of this method is that the Cake system is called 2 times for every request, however this can be alieviated with Caching. If you store the results of root_pages in the /tmp folder, you can build a caching system that will check to see if the file is outdated, and only load the requestAction() on such an occasion. Of course then you would store the file again.

Summary:

With this solution, you will not only have root level slugs (i.e. http://localhost/this-is-my-page) but also you will not need to modify routes every time you add a new controller to your system. The only disadvantage would be if a slug is called the same thing as your controller. In that case you simply decide ahead of time which has the priority, and the route will hit the first one that is decalred.

Links:

Learn more about Wildflower CMS.

New Direction for the Blog (CakePHP)

For the past year I have been developing nearly exclusively in CakePHP. If you haven't heard of it yet, there are still a few seats on the bandwagon. Feel free to hop on board.

As with any large scale application framework, learning the tricks of the trade is important. While there are plenty of blog sites out there, everyone comes across different problems and solutions.

Therefore, I am resurrecting this blog and I'm going to use it for CakePHP. Hopefully this blog will become a valuable resource for anyone wanting to learn Cake PHP, get tips on the best practices, etc.

I hope this is helpful for you.

Understanding Cake Routes

Cake Routes

What is a Route? Put simply, a route is a path from one point to another. In the world of networking, routes are built in routers to tell packets where to go to get to their destination. In the same way routing in Cake PHP is a set of rules that help a web browser display the right web page.

How do Cake Routes Work? Without exploring the 1000+ lines of code in the routes.php class, routes work by parsing the URL of the web page that you just requested. Your web browser sends that URL to the CakePHP Application. The Cake PHP application will then load your list of routes that you've defined.

Routes work by going Top-Down and looking for the first / best matching path. When it finds a match, the router ignores any additional routes that exist and it returns the routing information that decides which part of your application to load.

Why should we use routes? Let's say you have a giant application with tons of features, addons and tables. Now you need to load a single page. How does your application know just what files to load for this page without loading the entire application into memory? You could have an index.php file that contains a giant switch case with all the possible combinations of actions you have available in your system, however management of this becomes a nightmare. How will you determine just which classes and which files to load in a neat, managable, and expandable way?

Cake Routing is the answer. A Cake route connects a url path to a MVC Controller. That's it. Complete management in a few single lines of code. When the connection to the route is made, it will only ever load 1 controller. That controller decides which models (database tables) that it will be accessing along with which templates and layouts will be displayed. You never load anything that you don't need in that specific controller, which cuts down on memory consumption and helps the environment. Well not the environment so much, but it does cut down on needless hair loss.

Let's look at some examples:

Router::connect('/', array('controller' => 'pages', 'action' => 'display', 'home'));

The first route above does the following:

If the first thing I find after the URL is simply a "/" (which is usually always the home page) then we want to open up the pages_controller.php file in the app/controllers/ directory, and then look for a function called "display". Because "home" is stuck at the end of the array, it will pass the word "home" to that controller.

If it does not find /app/controllers/pages_controller.php, it will look in the cake core for the pages_controller (which is in cake/console/libs/templates/skel/controllers). If it can not find it there, cake will display an error.

Cake Routes also allow wildcards. This means that it will match everything. Here is an example:

Router::connect('/pages/*', array('controller' => 'pages', 'action' => 'display'));

This route does the following:

Any time there is a /pages/ after the URL, this route will kick in. It expects to have something after the pages/, and that something can be anything. For instance, if you had (http://url.com/pages/this-is-my-favorite-page), this route will kick in and it will send you to the pages controller, using the function display().

Once you get the hang of routes, there is quite a bit you can do with them. They can be customized in many ways.

Tips

• Remember that routes work from top to bottom. Place your most exact matching routes at the top, then less exact routes farther down.
• You can test routes by creating a file called "app/app_controller.php". Inside of it add the following lines:

1. function __construct() {
2. $route = Router::currentRoute();
3. pr($route);
4. parent::__construct();
5. }
   

With this, you can test different URLs, and this code will print out the route that it the URL matches.

Note: the pr() should be commented out when you are not testing routes.

• You can loop Route::connect commands in a foreach() statement to shorten the amount of code that you have. This helps for large administration pages.
  

Links:

You can read more about routes here.

Example routes can be found here.

It's been a while

It's been a while since I've posted. Since last year I've discovered CakePHP, which is an amazingly expandable and reusable framework. It is worth checking out if you plan to use for anything more than building simple static apps that never change.

Researching Development Frameworks

Researching frameworks can often times be a hassel. Often times PHP programmers need some sort of framework to be able to quickly deploy applications. So what is out there on the market? What is current, what is not?

The first one and what seems to be quite popular is Ruby on Rails, which is based on the Ruby language. Ruby on Rails is a web framework that creates fast web pages and code that favors convention over configuration. http://www.rubyonrails.org/ Also see http://www.ruby-lang.org/en/

It seems like a good idea for a number of back-end or fast applications. Similarly is a PHP Version called PHP on Trax (http://www.phpontrax.com). It comes as a PEAR module and is supposedly as quick to develop on as Ruby on Rails.

The next in the series is CakePHP. Cake is a rapid development framework for PHP which uses commonly known design patterns to rapidly develop robust web applications, without any loss to flexibility. (http://cakephp.org/)

These are all known in the design world as Model-view-controller (MVC) frameworks. They are based on real world problems and a need to rapidly deploy database applications without having to re-do all of the code. It separates the design model from the front end view and from the controller to create standards compliant applications with a good degree of flexability.

Here is a list of 10 frameworks.
http://www.phpit.net/article/ten-different-php-frameworks/1

Simple LAMP Setup

Installation

To install, we first need to extract each package. To do this, go into the directory containing the files you have just downloaded. Then type the following command:

# tar -xzvf apache_1.3.xx.tar.gz

# tar -xzvf mysql-3.23.xx.tar.gz

# tar -xzvf php-4.x.x.tar.gz

Next, We need to compile PHP and Apache. In this article we are going to opt for the fastest and best way to run PHP and Apache. That is by compiling them together into one executable. To do this, go into the apache directory and type the following:

# ./configure --prefix=/usr/local/apache

Next, Back out of the apache directory and go into the php directory. Then type the following:

# ./configure --with-mysql --with-xml --enable-track-vars --with-apache=../apache_1.3.xx

# make

# make install

Then, Back out again and go back into the apache directory and type:

# ./configure --prefix=/usr/local/apache --enable-module=rewrite --activate-module=src/modules/php4/libphp4.a

# make

# make install

This will compile and install Apache and PHP into the /usr/local/apache directory.

Now we need to compile and install the MySQL software. To do this, go into the mysql directory and type the following:

#./configure --prefix=/usr/local/mysql --localstatedir=/usr/local/mysql/data --disable-maintainer-mode --with-mysqld-user=mysql --enable-large-files -without-debug

# make

# make install

This will compile MySQL and install its files to /usr/local/mysql. Next, We need to create the user account 'mysql'. This user account is the account mysql runs as. To do this type the following:

# groupadd mysql

# useradd -g mysql mysql

Then we need to install the database files and make some minor ownership changes as follows:

# ./scripts/mysql_install_db

# chown -R root:mysql /usr/local/mysql

# chown -R mysql:mysql /usr/local/mysql/data

Then add the line to your '/etc/ld.so.conf' file:

/usr/local/mysql/lib/mysql

Next we run the mysql daemon. To do this, type the following:

# cd /usr/local/mysql/bin

# ./safe_mysqld --user=mysql &

Finally, we set the root password of the mysql database. To do this, type the following:

# ./mysqladmin -u root password new_password

We should now have Apache, PHP, and MySQL compiled and installed. We now just need to configure them.

Configuration

To configure PHP, copy the php.ini-dist from the php directory to '/usr/local/lib/php.ini'. This file contains most of the settings you would want but you may want to edit it. The one setting you might want to enable is 'register_globals' to 'On' since a lot of PHP scripts use global variables for form data. Other than you may leave it unchanged.

Next we configure Apache. To do this open the file '/usr/local/apache/conf/httpd.conf ' and add the following line right after the line 'AddType image/x-icon .ico':

AddType application/x-httpd-php .php

Then we start the Apache daemon. To do this, type the following:

# /usr/local/apache/bin/apachectl start

To test our PHP and Apache setup create a file in the '/usr/local/apache/htdocs' directory called test.php and put the following code:



Finally, Fire-up your web browser and point it to 'http://localhost/test.php' and you should see a php info webpage complete with settings of php.

Apache, MySQL, and PHP are all configured now and are ready to go!


Originally found here: http://www.sysbotz.com/articles/amp.htm

SCP without passwords

http://www.justenoughlinux.com/2004/04/14/scp_without_passwords.html
http://www.linuxgazette.com/node/193


In this article I'll show you how to use scp without passwords. Then I'll show you how to use this in two cool scripts. One script lets you copy a file to multiple linux boxes on your network and the other allows you to easily back up all your linux boxes.

If you're a linux sysadmin, you frequently need to copy files from one linux box to another. Or you need to distribute a file to multiple boxes. You could use ftp, but there are many advantages to using scp instead. Scp is much more secure than ftp, as scp travels across the LAN /WAN encrypted, while ftp uses clear text (even for passwords.

But what I like best about scp is that it's easily scriptable. Suppose you have a file that you need to distribute to 100 linux boxes. I'd rather write a script to do it than type 100 sets of copy commands. If you use ftp in your script it can get pretty messy, because each linux box you log into is going to ask for a password. But if you use scp in your script, you can set things up so the remote linux boxes don't ask for a password. Believe it or not, this is actually much more secure than using ftp!

Here's an example demonstrating the most basic syntax for scp. To copy a file named 'abc.tgz' from your local pc, to the /tmp dir of a remote pc called 'bozo' use:

scp abc.tgz root@bozo:/tmp

You will now be asked for bozo's root password. So we're not quite there yet. It's still asking for a password so it's not easily scriptable. To fix that, follow this one time procedure (then you can do endless "passwordless" scp copies):

1. Decide which user on the local machine will be using scp later on. Of course root gives you the most power, and that's how I personally have done it. I won't give you a lecture on the dangers of root here, so if you don't understand them, use a different user. Whatever you choose, log in as that user now for the rest of the procedure, and log in as that user when you use scp later on.

2. Generate a public / private key pair on the local machine. Say What? If you're not familiar with Public Key Cryptography, here's the 15 second explanation. In Public Key Cryptography, you generate a pair of mathematically related keys, one public and one private. Then you give your public key to anyone and everyone in the world, but you never ever give out your private key. The magic is in the mathematical makeup of the keys - anyone with your public key can encrypt a message with it, but only you can decrypt it with your private key. Anyway, the syntax to create the key pair is:

ssh-keygen -t rsa

3. In response you'll see:
"Generating public/private rsa key pair"
"Enter file in which to save the key ... "
Just hit enter to accept this.

4. In response you'll see:
"Enter passphrase (empty for no passphrase):"
You don't need a passphrase, so just hit enter twice.

5. In response you'll see:
"Your identification has been saved in ... "
"Your public key has been saved in ... "
Note the name and location of the public key just generated (it will always end in .pub).

6. Copy the public key just generated all your remote linux boxes. You can use scp or ftp or whatever to do the copy. Assuming your're using root (again see my warning in step 1. above), the key must be contained in the file /root/.ssh/authorized_keys (watch spelling!). Or if you are logging in as a user, e.g. clyde, it would be in /home/clyde/authorized_keys. Note that the authorized_keys file can contain keys from other PC's. So if the file already exists and contains text in it, you need to append the contents of your public key file to it.

That's it. Now with a little luck you should be able to scp a file to the remote box without using a password. So let's test it by trying our first example again. Copy a file named 'xyz.tgz' from your local pc, to the /tmp dir of a remote pc called 'bozo'

scp xyz.tgz root@bozo:/tmp

Wow !!! It copied with no password!!

A word about security before we go on. This local PC just became pretty powerful, since it now has access to all the remote PC's with only the one local password. So that one password better be very strong and well guarded.

Now for the fun part. Let's write a short script to copy a file called 'houdini' from the local PC to the /tmp dir of ten remote PC's, in ten different cities (with only 5 minutes work). Of course it would work just the same with 100 or 1000 PC's. Suppose the 10 PC's are called: brooklyn, oshkosh, paris, bejing, winslow, rio, gnome, miami, minsk and tokyo. Here's the script:

#!/bin/sh
for CITY in brooklyn oshkosh paris bejing winslow rio gnome miami minsk tokyo
do
scp houdini root@$CITY:/tmp
echo $CITY " is copied"
done

Works liek magic. With the echo line in the script you should be able to watch as each city is completed one after the next.

By the way, if you're new to shell scripting, here's a pretty good tutorial:
http://www.freeos.com/guides/lsst/.

As you may know, scp is just one part of the much broader ssh. Here's the cool part. When you followed my 6 stop procedure above, you also gained the ability sit at your local PC and execute any command you like on any of the remote PC's (without password of course!). Here's a simple example, to view the date & time on the remote PC brooklyn:

ssh brooklyn "date"

Now let's put these 2 concepts together for one final and seriously cool script. It's a down and dirty way to backup all your remote linux boxes. The example backs up the /home dir on each box. It's primitive compared to the abilities of commercial backup software, but you can't beat the price. Consider the fact that most commercial backup software charges licence fees for each machine you back. If you use such a package, instead of paying licence fees to back remote 100 PC's, you could use the script back the 100 PC's to one local PC. Then back the local PC to your commercial package and save the license fee for 99 PC's ! Anyway the script demostates the concepts so you can write you own to suit your situation. Just put this script in a cron job on your local PC (no script is required on the remote PC's). Please read the comments carefully, as they explain everything you need to know:

#!/bin/sh

# Variables are upper case for clarity

# before using the script you need to create a dir called '/tmp/backups' on each
# remote box & a dir called '/usr/backups' on the local box

# on this local PC
# Set the variable "DATE" & format the date cmd output to look pretty
#
DATE=$(date +%b%d)

# this 'for loop' has 3 separate functions

for CITY in brooklyn oshkosh paris bejing winslow rio gnome miami minsk tokyo
do

# remove tarball on remote box from the previous time the script ran # to avoid filling up your HD
# then echo it for troubleshooting
#
ssh -1 $CITY "rm -f /tmp/backups/*.tgz"
echo $CITY " old tarball removed"

# create a tarball of the /home dir on each remote box & put it in /tmp/backups
# name the tarball uniquely with the date & city name
#
ssh $CITY "tar -zcvpf /tmp/backups/$CITY.$DATE.tgz /home/"
echo $CITY " is tarred"

# copy the tarball just create from the remote box to the /usr/backups dir on
# the local box
#
scp root@$CITY:/tmp/backups/$CITY.$DATE.tgz /usr/backups
echo $CITY " is copied"

done

# the rest of the script is for error checking only, so it's optional:

# on this local PC
# create error file w todays date.
# If any box doesn't get backed, it gets written to this file
#
touch /u01/backup/scp_error_$DATE

for CITY in brooklyn oshkosh paris bejing winslow rio gnome miami minsk tokyo

do

# Check if tarball was copied to local box. If not write to error file
# note the use of '' which says do what's after it if what's before it is not # true
#
ls /u01/backup/$CITY.$DATE.tgz echo " $CITY did not copy" >> scp_error_$DATE

# Check if tarball can be opened w/o errors. If errors write to error file.
tar ztvf /u01/backup/$CITY.$DATE.tgz echo "tarball of $CITY is No Good" >> scp_error_$DATE

done

That's about it. In this article I've tried to give examples that demonstate the concepts, not necessarily to be use "as is". Some of the syntax may not work in all distros, but in the interest of brevity I could not include all the possibilities. For example, if you are using Red Hat 6.2 or before, the syntax will require some changes (I'd be happy to give it to you if you email me). So be creative and hopefully you can use some of this in your own environment.

Optimizing High Traffic Servers

Here is some very valuable information that I'm archiving.

Original URL: http://www.crucialparadigm.com/resources/tutorials/server-administration/optimize-tweak-high-traffic-servers-apache-load.php

Focus: Linux, Apache 1.3+, [PHP], [MySQL]
Notes: Use at your own risk. If this has any errors, please let me know and I will correct them.

Summary
If you are reaching the limits of your server running Apache serving a lot of dynamic content, you can either spend thousands on new equipment or reduce bloat to increase your server capacity by anywhere from 2 to 10 times. This article concentrates on important and poorly-documented ways of increasing capacity without additional hardware.

Problems
There are a few common things that can cause server load problems, and a thousand uncommon. Let's focus on the common:
Drive Swapping - too many processes (or runaway processes) using too much RAM
CPU - poorly optimized DB queries, poorly optimized code, runaway processes
Network - hardware limits, moron attacks

Solutions: The Obvious
Briefly, and for completeness, here are the most obvious solutions:

Use "TOP" and "PS axu" to check for processes that are using too much CPU or RAM.
Use "netstat -anp sort -u" to check for network problems.

Solutions: Apache's RAM Usage
First and most obvious, Apache processes use a ton a RAM. This minor issue becomes a major issue when you realize that after each process has done its job, the bloated process sits and spoon-feed data to the client, instead of moving on to bigger and better things. This is further compounded by a bit of essential info that should really be more common knowledge:

If you serve 100% static files with Apache, each httpd process will use around 2-3 megs of RAM.
If you serve 99% static files & 1% dynamic files with Apache, each httpd process will use from 3-20 megs of RAM (depending on your MOST complex dynamic page).

This occurs because a process grows to accommodate whatever it is serving, and NEVER decreases again unless that process happens to die. Quickly, unless you have very few dynamic pages and major traffic fluctuation, most of your httpd processes will take up an amount of RAM equal to the largest dynamic script on your system. A smart web server would deal with this automatically. As it is, you have a few options to manually improve RAM usage.

Reduce wasted processes by tweaking KeepAlive
This is a tradeoff. KeepAliveTimeout is the amount of time a process sits around doing nothing but taking up space. Those seconds add up in a HUGE way. But using KeepAlive can increase speed for both you and the client - disable KeepAlive and the serving of static files like images can be a lot slower. I think it's best to have KeepAlive on, and KeepAliveTimeout very low (like 1-2 seconds).

Limit total processes with MaxClients
If you use Apache to serve dynamic content, your simultaneous connections are severely limited. Exceed a certain number, and your system begins cannibalistic swapping, getting slower and slower until it dies. IMHO, a web server should automatically take steps to prevent this, but instead they seem to assume you have unlimited resources. Use trial & error to figure out how many Apache processes your server can handle, and set this value in MaxClients. Note: the Apache docs on this are misleading - if this limit is reached, clients are not "locked out", they are simply queued, and their access slows. Based on the value of MaxClients, you can estimate the values you need for StartServers, MinSpareServers, & MaxSpareServers.

Force processes to reset with MaxRequestsPerChild
Forcing your processes to die after a while makes them start over with low RAM usage, and this can reduce total memory usage in many situations. The less dynamic content you have, the more useful this will be. This is a game of catch-up, with your dynamic files constantly increasing total RAM usage, and restarting processes constantly reducing it. Experiment with MaxRequestsPerChild - even values as low as 20 may work well. But don't set it too low, because creating new processes does have overhead. You can figure out the best settings under load by examining "ps axu --sort:rss". A word of warning, using this is a bit like using heroin. The results can be impressive, but are NOT consistent - if the only way you can keep your server running is by tweaking this, you will eventually run into trouble. That being said, by tweaking MaxRequestsPerChild you may be able to increase MaxClients as much as 50%.

Apache Further Tweaking
For mixed purpose sites (say image galleries, download sites, etc.), you can often improve performance by running two different apache daemons on the same server. For example, we recently compiled apache to just serve up images (gifs,jpegs,png etc). This way for a site that has thousands of stock photos. We put both the main apache and the image apache on the same server and noticed a drop in load and ram usage. Consider a page had about 20-50 image calls -- the were all off-loaded to the stripped down apache, which could run 3x more servers with the same ram usage than the regular apache on the server.


Finally, think outside the box: replace or supplement Apache

Use a 2nd server
You can use a tiny, lightning fast server to handle static documents & images, and pass any more complicated requests on to Apache on the same machine. This way Apache won't tie up its multi-megabyte processes serving simple streams of bytes. You can have Apache only get used, for example, when a php script needs to be executed. Good options for this are:

TUX / "Red Hat Content Accelerator" - http://www.redhat.com/docs/manuals/tux/
kHTTPd - http://www.fenrus.demon.nl/
thttpd - http://www.acme.com/software/thttpd/

Try lingerd
Lingerd takes over the job of feeding bytes to the client after Apache has fetched the document, but requires kernel modification. Sounds pretty good, haven't tried it. lingerd - http://www.iagora.com/about/software/lingerd/

Use a proxy cache
A proxy cache can keep a duplicate copy of everything it gets from Apache, and serve the copy instead of bothering Apache with it. This has the benefit of also being able to cache dynamically generated pages, but it does add a bit of bloat.

Replace Apache completely
If you don't need all the features of Apache, simply replace it with something more scalable. Currently, the best options appear to be servers that use a non-blocking I/O technology and connect to all clients with the same process. That's right - only ONE process. The best include:

thttpd - http://www.acme.com/software/thttpd/
Caudium - http://caudium.net/index.html
Roxen - http://www.roxen.com/products/webserver/
Zeus ($$) - http://www.zeus.co.uk

Solutions: PHP's CPU & RAM Usage
Compiling PHP scripts is usually more expensive than running them. So why not use a simple tool that keeps them precompiled? I highly recommend Turck MMCache. Alternatives include PHP Accelerator, APC, & Zend Accelerator. You will see a speed increase of 2x-10x, simple as that. I have no stats on the RAM improvement at this time.

Solutions: Optimize Database Queries
This is covered in detail everywhere, so just keep in mind a few important notes: One bad query statement running often can bring your site to its knees. Two or three bad query statements don't perform much different than one. In other words, if you optimize one query you may not see any server-wide speed improvement. If you find & optimize ALL your bad queries you may suddenly see a 5x server speed improvement. The log-slow-queries feature of MySQL can be very helpful.

How to log slow queries:

# vi /etc/rc.d/init.d/mysqld

Find this line:
SAFE_MYSQLD_OPTIONS="--defaults-file=/etc/my.cnf"

change it to:
SAFE_MYSQLD_OPTIONS="--defaults-file=/etc/my.cnf --log-slow-queries=/var/log/slow-queries.log"

As you can see, we added the option of logging all slow queries to /var/log/slow-queries.log
Close and save mysqld. Shift + Z + Z

touch /var/log/slow-queries.log
chmod 644 /var/log/slow-queries.log

restart mysql
service myslqd restart
mysqld will log all slow queries to this file.

References
These sites contain additional, more well known methods for optimization.

Tuning Apache and PHP for Speed on Unix - http://php.weblogs.com/tuning_apache_unix
Getting maximum performance from MySQL - http://www.f3n.de/doku/mysql/manual_10.html
System Tuning Info for Linux Servers - http://people.redhat.com/alikins/system_tuning.html
mod_perl Performance Tuning (applies outside perl) - http://perl.apache.org/docs/1.0/guide/performance.html

Once again, if this has any errors or important omissions, please let me know and I will correct them.
If you experience a capacity increase on your server after trying the optimizations, let me know!

Article written by spagmoid additions by: albo,huck on the ev1 forums.