Nikolay Bachiyski gave a good talk at this year’s WordCamp San Francisco about unit testing WordPress plugins. His method uses PHPUnit, which is an additional PEAR package you have to install on the server. You can then use MockPress to simulate WordPress in the tests.
Another option is the SimpleTest for WordPress plugin, which uses SimpleTest instead of PHPUnit, doesn’t require installing a PEAR package, and lets you run both unit and integration tests.
Either way, you can use PHP’s ReflectionClass to test private and protected methods, although purists will tell you that’s not proper.