The core filesystem is built with scripts available here. The output is a tarball with no kernel, /opt/vc content or XBMC build. These components are fetched from the Raspbmc update system when booted with CDN synchronisation.
Building is simple, just run:
Ensure that you have debootstrap and the QEMU user emulation tools for chroot:
apt-get install qemu binfmt-support qemu-user
Needs to be the latest version, otherwise you will get weird errors. Latest version can be acquired from the sid repo of Debian.