I don't think I've ever had trouble with putting two special functions on top of one another. There may be something else causing the problem...
The workaround you mentioned would work, assuming that's the problem. And actually, you could get 4 functions in a space of two by two tiles.
I've tried making a menu without script. Yeah, it can be done, but it gets pretty hairy. You just need to make sure you're keeping up with all those functions. One suggestion: instead of using three special functions per tile, make just one for selecting per tile, and then two tall ones for the "move up" and "move down" functions. Make sure they overlap on the right tiles, that is, make sure you can't go up when you're at the top option.
Hope that helps.