Overclocking is the art of setting your hardware to run faster than the default manufacturer setting. Overclocking your system can be a fun and rewarding activity but it comes with some risks. In fact there is no guarantee that your system is even overclockable. The hardware you bought is only guaranteed for the speed printed on the packaging and anything faster is purely a bonus. Just because someone else on the internet managed to get their hardware overclocked so fast it went back in time, doesn't mean you can. Overclocking can also make your system unstable, leading to hard lockups and software or OS crashes. Overclocking may also void your warranty.
First we need to know how the system works in order to manipulate it. Most PCs work on a reference clock system, also known as the frontside bus (FSB). Everything on the motherboard works off of this clock such as the CPU, RAM, PCI slots, etc. Because not everything runs at the same speed, the different subsystems use either a multiplier or a divisor. For example the PCI bus runs at 33MHz so if you have a 200MHz bus you need a divisor of 6 (200 / 6 = 33). If your CPU runs at 2500MHz you need a multiplier of 12.5 (200 * 12.5 = 2500). So by altering bus speed and/or multiplier and divisor settings we can adjust the speed of your system.
The next generation of hardware appears to be moving away from the frontside bus method and moving to a more efficient HyperTransport system. This is a more point-to-point link system and allows more control over each system independently. Which is a good thing if your an overclocker.
Heat is Your Enemy
Overclocking your system will cause components to work harder and draw more power leading to more heat being given off. If your components get too hot they will become unstable and lead to system crashes or lockups. Stock heatsinks and thermal paste usually aren't very good for overclocking. If you want to max out your performance you may need to look into high performance heatsinks and high quality thermal paste like Arctic Silver 5. Below is a small table I got from Wikipedia that shows how well certain materials are at conducting heat.
Thermal conductivity @ 25C
Before starting make sure you have the latest BIOS flashed for your motherboard. If your not familiar with how to reset your BIOS settings be sure to have your motherboard's manual handy as well. Also have some way to benchmark your system before you begin and after you make changes. Some people "overclock" their system only to find it perform slower or about the same while sacrificing stability.
The simplest and most stable way to overclock a system is the CPU multiplier way. This only effects your CPU which means maximum system stability. Unfortunately both AMD and Intel started locking the multiplier on most of their chips several years ago because too many people were buying the cheaper CPU's and overclocking them to a more expensive version. A few of their higher end CPUs, like the AMD Black Editions, are advertised as multiplier unlocked so Google around to find out if your processor is unlocked. To use this method simply enter your BIOS and bump up the CPU multiplier by 0.5 then save and reboot. If the machine reboots, you are good to go and can continue to bump it up by 0.5 until you get the "black screen of death". Most motherboards will recover after 10 seconds or so and reset your BIOS settings back to default. If nothing happens you will have to pull the battery and/or use the CMOS reset jumper on your motherboard. Then re-enter the BIOS and set it back to the last working setting.
The second, more common way to overclock your system is FSB overclocking. This method overclocks everything on your motherboard which can be beneficial but risky. While increasing the speed of your RAM can be good, increasing the speed of your PCI or PCI-e busses can be bad. Some motherboards have the ability to lock the speed of your PCI-e bus, chipset, and/or RAM (recommended). To use the FSB method, enter your BIOS and bump your bus speed by 5-10MHz and then save and quit. Do this until you get the "black screen of death" or your motherboard throws an error then renter the BIOS and select the last working speed.
Testing & Voltage Tweaking
Just because your motherboard POSTs does not guarantee that your system will be stable however. You must boot into your OS and do a "burn-in" test. Use a benchmarking tool, like Sandra, and do several runs or launch a game and play for awhile. If the program or OS crashes you can do one of two things. The safe way is to just go down another 0.5 if you did the multiplier way or 1MHz for the bus way and try again. You may also be running into the problem of where you are overclocking the PCI/PCI-e bus too much and it has become unstable. To fix this some motherboards allow manual adjustment of the various divisors while some others will try to auto adjust. Usually divisors are whole numbers only so this can potentially lead to underclocking some systems.
The dangerous way is to slightly raise the voltage of the component that is holding the system back. You can first try adjusting the RAM voltage. The normal voltage for DDR2 is usually 1.8 to 2.0 volts. To get more speed or stability you may need to push it up to 2.1 or even as high as 2.2 volts. Anything above 2.2v is very dangerous and I wouldn't even recommend 2.2v unless the RAM has good heatspreaders. Move up in the smallest increments possible and if you don't get any more speed be sure to set the voltage back to the default value.
Next you can try boosting the CPU voltage (Vcore). Slowly increase the voltage by 0.025 volts (or whatever the minimum increment is) and see if you can get a bit more speed out of the system. Be careful about going more than 0.15 volts above the default.
Video Card Overclocking
If you have a fast and stable system but your just looking for a few extra FPS in your favorite games you can try video card overclocking. Because there is no direct hardware access you'll have to download a software utility. RivaTuner is a popular one but many others exist and some card manufacturers actually include one on their website. Using these tools you can adjust the various clocks on your video card. If you begin to push your card too hard you will start to experience video artifacts like static or flashing polygons.
Have Fun and Remember
A few extra percent is usually not worth the instability. Different games/programs will push different parts of your system to the limit. Just because the couple programs you tested before were stable doesn't mean another program will come along and cause trouble. If you're experiencing any sort of instability set your system back to default and see if it goes away. You may have to make adjustments in the future.