A bit is short for binary digit. That is, they are digits of a base 2 (binary) number. Whereas the normal base 10 (decimal) digits are numbers between 0 and 9 (inclusive), binary digits are either 0 or 1.
A byte is the smallest addressable unit of memory in a computer systems. In practically all modern computers one byte consists of 8 bits but historically this has varied from 5 to 12 based on how many bits were needed to represent one character from the character set of the computer.