Sensors are just devices which 'sense'. Nothing more..
They are just transducers which convert some quantity x into another quantity y.
By measuring y or the change in y, you can correspondingly deduce x or the change in x, as the case may be.
As to what sensors to use in your robot, that is also largely your choice.. Depending on the requirements for the robot, your proficiency with a particular kind of sensor and some other things like cost and ease of use, you can select your sensor.
For example, in your line following robot, you can use a IR transmitter and receiver or a photoresistor and a LED, or even a laser sensor if you want that much accuracy.
Similarly for a robot to sense a cliff, same thing can be used. Use an IR sensor. If you're on the floor, you'll get a reading from the sensor. If you go to a cliff and are about to fall, sensor output will be zero. Use that..
All the four cases you mentioned in the post are instances where something called "Proximity sensors" are used. Just google it and you'll get a pretty good idea about it and how to use it in your robot too.