if you have some knowledge of unix you will find that on a vt100 terminal, it is not possible to draw graphics.
graphic libraries is only available for X. you can find some information about Xlib. now there is SDL libarary, it may help you also.
but what you want is to build a graphic application with out X, you will have a hard time. You need to learn a lot of hardware specific things. and even you manage to built one on your own machine, it may not run under other system.