I bought an ES1868-based sound card and the game port on it had issues with the analog input bouncing between 0 and the mid point for both the X and Y axis. These issues are cleared up when I have your USB keyboard and mouse drivers installed, which both work great. I haven't actually tried the USB joystick drivers yet.
I've tested it both ways a number of times now, when I don't have your driver loaded the game port jitters and when I have your driver loaded it works exactly as it should. I'm rather curious what your driver might be doing to resolve the game port issue as I'm not sure what's causing it to begin with nor do I see how a USB driver would affect just an ISA sound card's game port. Before your driver loads I use BIOS legacy USB support which only provides keyboard emulation on my PC, no mouse.