1) ATX and BTX are specification standards. The direct way is to find what the standards specify and verify that aginst the board's properties. However, that is unnecessarily lengthy. (For the specs, see Wikipedia entry on ATX and BTX, and the board's instruction manual.
2) It's usually written in the board's manual, as well as in the shop entry in most specialized IT shops.
3) To do-it-yourself: Measure the board. (Lenght x width). That's the most apparent difference (but not the only one!). ATX boards are 305 mm × 244 mm. BTX is 325×266 mm. So, you're gonna need a precise measurement (the difference is 2 cm on each side).
As an addendum: ATX and BTX boards are only partially intercompatible (which is meant to say, not all ATX components are BTX-compatible and vice versa). Also, not all cases support both formats. Finally, BTX is quite rare today, as it never really established itself as a common form factor (form factor means the size spec).
If you're interested in the topic, check out http://en.wikipedia.org/wiki/BTX_(form_factor) , it links to all the other form factors as well. Also go see a WTX board. Those are HUGE! (Physically, I mean. :-) )