With feedback, we can do very much better and achieve a nearly perfect voltage source. We can see this principle which is shown in the circuit below. Using a voltage divider on the output or the voltage reference, the output voltage is compared with an accurate voltage source such as the 2.5V LM336 to get the desired voltage and the difference drives an operational amplifier that acts to reduce it to zero by controlling a pass transistor.
Current limiting feature is available on this circuit which protecs against a shorted output. Q2 draws extra current from the op-amp output when the voltage across the current-sensing 33 Ohm resistor rises sufficiently to turn on Q2 (about 0.7 V). The output voltage is tried to be maintained by the op amp, but eventually saturates and then the output voltage can drop. The 1K resistor is used to help the amplifier to saturate when this happens (it would saturate anyway, since the output can only source about 20 mA, but the resistor does it more gently). Practically, all advanced voltage regulators include current limiting as good as other protection against unfortunate events.