How does security and encryption defeat someone just booting into recovery?

Regardless of device encryption and fingerprint scanners and the like, couldn't someone just get your phone, boot into recovery, and factory reset it? Wouldn't that undo any security you had on device?