That was my idea too, and why I started working on a light fork of pkgsrc to try it. Sadly, nobody who has to maintain systems at scale will use a package manager that just pulls random repos and "makes and prays". I've been trying to find the middle ground since then but haven't come up with it yet.
MacPorts? I was using NetBSD pkgsrc, which does either source builds or packages from the same system. So, you can lock it at a particular source revision/release and make an official package like a .deb, or do a gentoo/MacPorts style source build.
As I understand it, Arch's pkgbuilds are similar. The nice thing is that they have the flexibility to do whatever you want, whether it be updating a svn repo, downloading a binary, curling a script, building from source, whatever.