It appeared to be my OGL calls. Since I left an increasing amount of garbage each frame, mainly from drawing 2 spheres each frame in immediate mode, and the GC didn't have time to dispose what I had used faster than it was made,
System.Threading.Thread.Sleep(0);
This does the trick. It gives the GC enough time to breathe. I've stabilized the Mem to an even 137,000, which, mind you, isn't entirely impressive, compared to other games I play which are a clean 40,000 to 45,000, it's much better than a 200,000 game (Hey, even look at Minecraft- but I haven't played that or updated it in 3 months). Especially since I have so much needing to be drawn in Immediate Mode, and Lidgren.Network doesn't seem to have much of an effect on the Private Working Set of memory.
The game had even crashed my PC about three times before. Things became slow, I checked the processes- about 2,400,000 K :O and for some reason the Apache Handler on my computer had said "The process completed successfully" and I had an unreadable memory error. Before, it happened with Google Chrome (The flash player was unstable) but I'm really hoping the game can 'play nice' around other processes without 'ruining it for everybody'