Why not voxels?

I've been hacking around in Rust and OpenGL for a few months now, working on a Minecraft-inspired, procworld-inspired "game" I've dubbed playform (a working title - if you have better ideas, let me know). When I started this project, I was a big fan of voxels, and my love grew with advancements like this one, wherein I saved a ton of VRAM (my bottleneck at the time) by just sending positions of blocks, rather than the position of each vertex of each triangle (at 12 triangle per block, that's 36 times more VRAM!)

But with time, research, and careful re-re-reading of procworld posts, I've come to think that voxels are a lot less magical than I once thought.

First of all, it's worth noting that you don't just have to have one data structure in your game. For everything to work as well as it can, you'll probably have lots of different ones. You might treat everything as a box for physics purposes, but that doesn't mean you need to render boxes; they can be slanty or curvy on the screen, without taking that into account anywhere else. Just because your UI makes all the blocks grid-aligned doesn't mean you have to enforce that throughout your entire codebase, and in fact, doing that would probably do more harm than good, since you're going to now have a bunch of special casing for all the things that aren't grid-aligned, like mobs and projectiles.

As far as rendering's concerned, voxels are just a form of compression - if you're going to only be rendering cubes, why would you send all that triangle data to VRAM? You can take advantage of the fact that all your shapes are the same, and just send the parts that matter. But like all compression, it comes at a cost: namely, how quickly you can use the uncompressed data. Like it or not, the graphics card renders triangles, and you can either send it the triangles directly, or send it blocks that will turn into those triangles, but it'll render the same triangles either way. It'll do it quicker if you just hand them to it directly, instead of making it jump through hoops. So if you can fit all the triangles you want to render into VRAM uncompressed, the compression will only cost you. The question then becomes whether you can fit all those triangles into VRAM, and how much work it will take to only keep the relevant ones in VRAM.

There's also the issue of how fast it is to send uncompressed vs compressed data to VRAM, but that seems relatively minor - initial loading time is a fairly minimal thing to be considering, since it only happens once every time you initialize a scene, and streaming the updates after that seems like a fairly solved problem too, based on my (admittedly limited) research.

And of course, having arbitrary triangles gives you tons of advantages too - you can draw whatever you like, however you like it.

So what are voxels good for? Well, as this procworld post points out, procedural generation is a lot easier when you're making things on a grid. And as Minecraft demonstrated, it makes the UI a breeze for malleable terrain. But like I said, you don't need to have a single data structure in your game; use each one only where appropriate. Once you've gotten the advantages of one data structure, it's fine to convert your results to something that's more useful, like turning your voxels from your easier terrain generation into a triangle mesh for rendering. Your physics can still treat things like boxes. Your UI can still mostly add and remove blocks. But your render code doesn't have to think the same way the rest of it does, and that also gives you the freedom to make your UI a little more free, too - maybe you want to add slope to your blocks, or subdivide them. If your whole game is predicated on the idea of a rigid grid with just a block type in each cell, and every part of your "engine" from terrain gen to rendering relies on that idea, then making those changes is going to be really obnoxious.

Obviously, there's a cost to converting between data structures, but don't forget the option entirely; just weight it appropriately against the others.