The audacious plugin is well out of date. There was an effort to port it to 3.x here, you might just be able to take unix/, configure.ac, readme.txt from eppixx's repo, but I've certainly not tried it myself.
edited 8:36 PM EDT April 17, 2017
Made some (unsuccessful) attempts, for the current vgmstream you're going to have to figure out how to deal with the external libraries, they're not trivial to turn off with #defines anymore.
audacious/util.h is a header from 1.x, it no longer exists. If you've installed the audacious-dev package (on Debian or Ubuntu) the configure script will find the audacious dir on your system, and the 3.x code from eppixx shouldn't use audacious/util.h anymore.
@Bladeforce - I suspect you are trying to build the old SVN vgmstream (there is no main.c anymore). Latest vgmstream: https://github.com/kode54/vgmstream
The current Audacious plugin version was fixed by TricksterGuy based on eppixx's changes. I further did minor tweaks and added build instructions to my (admittedly poorly written) BUILD.md, just now.
I tested it in Linux Mint and should work fine for Audacious >= 3.5+ (not older, as them Audacious devs can't have a consistent plugin API). Audacious 3.8 may not work since they changed the API again but I can't test it right now (I did some blind fixes tho). Works with OGG/Vorbis and MPEG, but not FFmpeg (ATRAC3, XMA et al).