Very simple. Either life outside of earth doesn’t exist or we’ll only see it if we cheat.
The universe could be filled with tons and tons of planets brimming with life. We very well could live and die as a species without ever knowing. Stars are really far apart. Words can’t accurately described the distance; we make do with scientific notation. You can’t travel at light speed, or a fraction of light speed. Relativistic speed is a Bad Thing. Ignoring time issues, it takes an extraordinarily large amount of energy to get to those speeds. If you can possibly get to those speeds, you will eventually hit something. Doesn’t matter what it is, could be the size of a pea. At relativistic speeds, that’s more or less like running into a small planet or whatnot. But let’s say you COULD go the speed of light, safely. Ok, you could realistically visit a handful of nearby stars with very big starships that were multi-generational. Not a bad solution. But the probable lifespan of humanity would be limited to less a dozen solar systems, none of which could probably be reached in one lifespan. There are not more than a handful of stars within a million light years of earth. A million years is a bit long for any piece of technology to survive. Entropy is just one of those facts of life.
It doesn’t matter if you could go the speed of light, ten times the speed of light or a thousand times the speed of light. You’d be limited to a very, very small corner of the universe. The only way to realistically travel the universe would be near instantaneous travel. It’s not faster than light (FTL), it’s instantaneous or nearly so. Space folding, wormholes, alternative reality hacking, whatnot.
It may be very possible that instantaneous travel across astronomic distances is impossible. Current physics don’t seem to think it’s practical. So when someone acts smug and quotes Fermi paradox, remind them that Fermi was ignoring the distance factor. Same with the Great Silence. A civilization on a distant planet could turn their entire planet into a huge radio and we wouldn’t notice it for millions, hundreds of millions or billions of years. The universe is spreading out at a rapid distance. Each second, most galaxies move a bit further away.