8 Replies - 903 Views - Last Post: 25 February 2016 - 08:51 PM Rate Topic: -----

#1 kutuup  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 30
  • Joined: 01-August 14

Struggling to understand how graphics work in terms of code.

Posted 24 February 2016 - 05:27 PM

I've watched a bunch of videos and read a ton of material on this, but I can't wrap my head around how graphics work in terms of code.

To be clear, I get how graphics work and are manipulated, it's specifically how the code behind processes such as drawing pixels to the screen and generating a frame works.

I have a rudimentary understanding of OpenGL, in that I can draw shapes with it, do some basic manipulation on them etc. beyond that, my knowledge is sorely lacking.

I'm not looking for you to teach me how to program graphics engines, I'm just struggling with some core concepts.

For example, I've pulled apart the source code for Doom, and I can identify WHERE they are doing particular things, but I don't see any code in there that actually draws to the screen. You have functions for drawing vertical stacks of pixels, or horizontal bars, but the functions don't actually seem to draw anything to a buffer or similar structure to be "fired" at the screen, they seem to just calculate where the pixels should be.

I think a part of my problem is presuming that things are WAY more complicated than they really are when you understand them, so I get daunted and kind of shy away.

For example, in the Doom engine, I THINK the way they draw the maps is via custom data structures that describe the walls as 2D lines with a height, and when the game renders the wall, it basically grabs each point along the line and draws pixels vertically from said point until it reaches the defined height, then moves onto the next point.

Is it really that simple? Surely it can't be.

Even if it is that simple, I don't see how they're drawing to the screen in the first place. I mean, that game was made for DOS (if I recall correctly). In those days, there was no such thing as a window in which you can draw something, so what exactly were they using to grab control and manipulate the display? Did they write everything from scratch? Moreover, how was it possible for one guy to do that in less than a year?

Is This A Good Question/Topic? 0
  • +

Replies To: Struggling to understand how graphics work in terms of code.

#2 modi123_1  Icon User is offline

  • Suitor #2
  • member icon



Reputation: 13392
  • View blog
  • Posts: 53,445
  • Joined: 12-June 08

Re: Struggling to understand how graphics work in terms of code.

Posted 24 February 2016 - 05:37 PM

I haven't been in the Doom engine, but typically - for 3d games - there are vertices that make up a polygon.. then textures and what not are applied to that polygon and they continue on.

https://www.iddevnet...ngame_step1.png
https://glumpy.githu...gl-pipeline.png
Was This Post Helpful? 0
  • +
  • -

#3 kutuup  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 30
  • Joined: 01-August 14

Re: Struggling to understand how graphics work in terms of code.

Posted 24 February 2016 - 06:46 PM

View Postmodi123_1, on 24 February 2016 - 05:37 PM, said:

I haven't been in the Doom engine, but typically - for 3d games - there are vertices that make up a polygon.. then textures and what not are applied to that polygon and they continue on.

https://www.iddevnet...ngame_step1.png
https://glumpy.githu...gl-pipeline.png


That's generally how it works in true 3D engines as far as I've seen, but the Doom engine isn't 3D, more 2.5D. It operates in 2 dimensions, but is drawn to appear 3D.
Was This Post Helpful? 0
  • +
  • -

#4 BBeck  Icon User is offline

  • Here to help.
  • member icon


Reputation: 792
  • View blog
  • Posts: 1,886
  • Joined: 24-April 12

Re: Struggling to understand how graphics work in terms of code.

Posted 25 February 2016 - 08:10 AM

So, I should say that I've never pulled the Doom engine apart, first of all. But as you say, it's 2.5D; I've read that they cheated and made it more 2D than 3D because of the graphics capabilities of the time.

One thing you have to understand is that the graphics hardware has changed substantially over the last 3 decades. Code that would have worked at the time will not even run on today's hardware and operating systems.

Pretty much everything comes down to writing to a buffer with any graphics I've ever seen.

And it's been forever since I've played Doom. So, I got on YouTube to remember what Doom looks like. It appears to be 3D, rather than 2.5D; I believe where the "2.5D" idea comes from is that it's more like 2.9D in that it uses billboarding pretty heavily. So, monsters that appear to be 3D, may actually be 3D quads displaying a 2D sprite that always faces the character. So, the walls and such are actually 3D. There's also a skybox.

Wolfenstine appears to be the same. In fact, the floors vary in height.

The games came out in the early 1990's for PC. This appears to be before DirectX even existed.

So, at that time every video game that was produced had to basically write it's own version of "DirectX". It was the DOS days and every computer out there ran a different graphics card that ran in an entirely different way. So, you were programming to say 20 different types of graphics methodologies simultaneously. There would not have been a standard buffer that you could write to. At best, you could be writing to your own standard buffer and blit'ing it to the various buffers you need to write it to. Things were kind of a mess. If the game didn't code for your exact graphics card, you could not run the game. There were only a hand full of graphics cards the game would run on. And the programmers had to code specifically for each graphics card.

So, no telling how they coded that. It would depend a lot on how the graphics card you are talking about worked. Having some knowledge of DirectX9, I suspect that if the graphics card was 3D, it might actually have you specify quads and a texture rather than writing to a buffer or even drawing triangles. Graphics cards used to do a lot more for you than today's graphics cards do. If they didn't you might actually have to do your own 3D to 2D conversion, which can get ugly and complicated. I've gone through the specifics on how to do that but it's mostly irrelevant today because the graphics hardware today isn't really setup to allow you to do that, per sae.

Anyway, the graphics hardware today just flat out doesn't work that way anymore. First you had DirectX and then DirectX 9 where Windows took away all your ability to talk to the graphics card directly and gave you an interface to talk to the graphics card through. So, you can't talk directly to the graphics card today even if you wanted to without getting rid of Windows and using a different operating system. (I'm not sure any operating system today lets you talk directly to the graphics card like back in the DOS days, for one thing because it was disastrous. You had to know how to code for 20 different graphics cards and any graphics card you didn't code for would not work.)

DirectX 9 still did a lot for you. There was a built in model class, for example. I think there were built in lights. I never got too deep into it because I really started understanding how this stuff works in the DirectX 11 days and quickly realized that DirectX 11 is as different from DX9 as it is from OGL. While they share a few things in common, even the hardware for these frameworks changed. For example, a DX9 graphics card will not run DX11. I think DX11 will run on a DX9 graphics card with greatly reduced features. But even then, I don't think you code it with DX9 code, for the most part.

OGL is a bit different than DX. I've been meaning to learn OGL and just haven't had the time. I may be getting some of my time back here soon and may be able to get back to it. But I believe Doom even precedes OGL by a year or two or so. So, even OGL 1.0 may work entirely differently than Doom graphics. And I guarantee OGL 4 does.

I think the bottom line is that if you are going to write Doom for modern graphics cards, you are going to have to write it substantially different than the way the original was written. The old code won't run on Windows, because it's written for DOS and graphics cards that no one has anymore. Today's graphics cards have to work the way OGL and DX tell them to (or rather the way their standards say the graphics card must adhere to). You no longer care how the individual hardware works, and it all conforms to the standard anyway. Plus, more recent versions of OGL and DX get you pretty darn close to the metal anyway.

The bottom line is that I have no idea how Doom was coded, but there's only one way to code it now. And that's with vertices and triangles going through a vertex and possibly an index buffer. If you want to code it on today's computers, I can tell you the only way it can be coded because there is only one way now on today's hardware.

While I haven't got deep enough into OGL to tell you how to code that, my graphics series in on HLSL which is nearly identical to GLSL. So, I can say with confidence that my YouTube graphics series shows you how it's done in the more recent versions of OGL and how it's done on all modern graphics hardware. The hardware itself no longer supports any other method.

Even 2D games must be coded this way in DX11 and presumably in the most recent versions of OGL. If older versions of DX or OGL will even run on the graphics card, there is likely a conversion to this somewhere in the pipeline. This is the direction that all graphics hardware is going in where it no longer truly supports 2D and everything is done this way.

Depending on your background, you may want to start with my vector and matrices videos. If you feel very confident with those two subjects, then you can get directly into the HLSL stuff. You can probably skip the Gimbal Lock video. You might be able to skip straight to the graphics theory with the second video in the HLSL series. If you skip the first video in the HLSL series, it covers mostly the XNA code that calls the HLSL shader, which would be different if you are using C++ although the HLSL is the same for DX and would be nearly the same in GLSL and the theory would be identical. So, the series is a good tutorial for anyone coding in DX11 or GLSL as well. The graphics card does it's thing in the same way no matter which you are using and you are so close to the metal at this point that all graphics today work this way no matter what you are coding with.

So, for Doom today, I would make the building and such fully 3D with various section models snapped together to form a level. I'm learning more about this by doing level design for Skyrim using the Skyrim Creation Kit. That's a pretty good education in level design for 3D games. I'm learning quite a bit from it. Then you can take those ideas back to your own game engine. For the monsters, I would probably make them "3D sprites". So, they would actually be billboards which are rectangular quads that always face the camera and display a 2D sprite in a 3D world. The HUD would be quads that are in screen space that likewise draw 2D images on them.

On today's graphics cards, there's really no other way. At best, your graphics card may support some older versions of OGL or DX that support some older methods of drawing. Even then, you would likely have to do it the same way; the difference being that it might have a sprite class that will draw the 2D stuff for you. But it would still pretty much be a 3D world. Even the original appears to be fully 3D with billboards that you might be able to think of as 2D.

This post has been edited by BBeck: 25 February 2016 - 08:14 AM

Was This Post Helpful? 0
  • +
  • -

#5 BBeck  Icon User is offline

  • Here to help.
  • member icon


Reputation: 792
  • View blog
  • Posts: 1,886
  • Joined: 24-April 12

Re: Struggling to understand how graphics work in terms of code.

Posted 25 February 2016 - 08:59 AM

Another thing that I might mention is that the way it all works in theory. It all starts with points in 3D space. You can project those points in 3D space onto a 2D surface.

So, you have your 3D points in memory, but you want to see them on a 3D screen. First, you need a camera into the imaginary 3D world. For the simple camera, imagine a rectangular 3D box aligned with the three 3D axes. We can now capture all the points in the 3D box and remove their Z component to squish them all onto the XY plane. You then have 2D data that you can draw on a 2D screen. And if you rotate the points, they will appear to rotate on the 2D screen because their relationship to one another will change in that way. You will still be able to see their relative positions in 2D. So, you have a 2D window into your imaginary 3D world.

Now you can project the 3D points to a 2D surface, and then draw lines between them. This allows you to make wireframe 3D models that you can view on a 2D screen but appear 3D.

You have orthographic projection at this point and have no perspective. Perspective projection is more complicated, but works basically the same way. Instead of a 3D box, you use something more like a 3D four sided pyramid and capture all the 3D points within that area. You project them down to the tip of the pyramid and before they get there you have a 2D plane that the points project on to (before all the points merge into a single point at the tip). The points get closer and closer as they approach the tip of the pyramid (the pyramid is on it's side remember).

And then once you have the points projected onto a 2D surface you can then draw lines between them and then draw that on a 2D video screen. Now you have perspective and things truly begin to look 3D.

But this doesn't give us "solid" objects. We just have wireframe meshes at this point.

So, the next step is to use some geometry here and use the points to form 3D triangles in 3D space. When we project the 3D points of our triangles to the 2D screen, we can then draw lines between each of the 3 points in every triangle and form objects out of 3D triangle meshes. The reason this is important is because of the magic of triangles.

The next step we need to do is to fill in the triangles or "shade" them to make them solid. We have our 3D triangles on our 2D surface with the lines drawn between them. When you move them in 3D they appear to be 3D on the 2D screen. But on the 2D screen they are actually 2D triangles. Still, we "know" exactly which pixels on the 2D screen are "inside" the lines of our triangle on the screen. We know which pixels on the 2D screen are in bounds for the triangle and which are out of bounds. So, we can just light all those pixels inside the triangle.

It should be noted that on modern hardware you really light pixels yourself. This is the job of the rasterizer which is built into your graphics card. I don't think that even DX12 allows you to take over the rasterizer and do your own thing, but it's not necessary because your pixel shader tells the rasterizer how to color each pixel. Even then, the pixel shader tells the rasterizer the color of every pixel but the rasterizer still has the job of lighting the pixels.

Now, if you watch my HLSL series, you'll see that what you have at this point is a colored silhouette. What you have at this point is more like a shadow of a model than looking like an actual model. I'll leave it to the series to go into further depth on where you go from there, but this is the basic theory of how all graphics works today.

You still have a lot of problems to address at this point though. One is Z-buffering. Your models will draw on top of one another according to their draw order rather than their distance from the camera and seriously mess up the illusion of being 3D. (The graphics card, again, mostly handles this for you today although you have control over these settings. But there was probably a time when you had to code this sort of thing yourself, probably back in the Doom days.)

You use surface or vertex normals in order to do lighting calculations to make your model into something that really looks 3D instead of just a silhouette. I'll leave the explanation of that to the videos.

Then instead of shading in the triangles with just a color or a color gradient to simulate shape, you can shade the triangles in with the texels of a photograph (basically pixels of a photograph). And you can stretch that photograph across the triangle by mapping points in the rectangular 2D image to the three 3D vertices of the triangle using what we call UV coordinates.

Now your models look much more realistic, although we're at this point probably still talking about late 1990's graphics here.

From there you instead of using an image to simply wrap the model in, you can use an additional image to store surface normals. This allows you to specify a vector normal for every pixel of the triangle rather than just at the corners. So, every pixel can face a different direction for lighting purposes. This gets into bump mapping, normal mapping, and parallax mapping and adds a lot more authenticity to the model. Here basically what you are doing is taking a model with a limited number of vertices, say 2,000 vertices, and then when you sculpt it you push the vertex count up to say a million vertices and do your sculpting to the point where it's starting to look as real as you can imagine. Then you record all that millions of vertices information by projecting it back to the 2,000 vertex model and record the facing of those extra vertices in the normal map (or maybe today in a parallax map). So, now your 2,000 vertex model looks like a million vertex model because of normal mapping. The modeler probably actually sculpted on a million vertex model and then compressed it all down into a normal map or parallax map. But the graphics card is working with a 2,000 vertex model, which is a walk in the park for the graphics card compared to processing a million vertices. The shader is fed the normal map and draws the individual pixels on the surface of the triangle with facing (normals) information from the normal map that was made with a million vertex model and so it has the look of a million vertex model even though it's actually only 2,000 vertices. This is highly efficient for the graphics card. The normal map is produced in your modeling software such as Blender.

Then there's new tricks on top of that. There's all kinds of maps that are basically just textures that work for the most part the same way that texturing works. The normal map maps across the model exactly like a texture. But the data in the map and how it's used is what makes the difference. There's several other types of maps that allow you to increase the authenticity of your model including specular maps that allow parts of the model to reflect light while others do not and maps that allow you to do ray tracing before runtime and record that information into a map for the advantages of that lighting without the cost of it at runtime.

Today you also have geometry shaders and tessellation shaders that take things even further.

But even 2D today is done this same way, although you're probably not going to use much more than two triangles as a quad and a texture drawn on the quad as a sprite and be done with it.
Was This Post Helpful? 1
  • +
  • -

#6 BBeck  Icon User is offline

  • Here to help.
  • member icon


Reputation: 792
  • View blog
  • Posts: 1,886
  • Joined: 24-April 12

Re: Struggling to understand how graphics work in terms of code.

Posted 25 February 2016 - 09:10 AM

Oh! I forgot to mention how the camera works. What I described so far gives us a very realistic 3D world with a camera that is stuck and cannot be moved because it has to be looking straight down the Z-axis in order to do it's job. The math otherwise will not work for the projection.

That's where the view matrix comes in.

I hadn't mentioned matrices. But you use a world matrix to store the position, orientation, and scale of every 3D model. This allows you to move them around the scene independently without changing their vertices and potentially destroying the model, for one thing.

You can store a view matrix for the entire world. (One view matrix per camera.) And this is used to move the entire world around the camera. It's all relative. If the entire world moves in a uniform way, it will appear that the camera is moving rather than the world moving. The user will think the camera is moving through the scene, when in reality the scene is moving around the camera. What ever position and orientation is stored in the view matrix, you move everything in the world by that much so that it is oriented in that way.

The problem is that this is backwards. So, if you rotate the world clockwise, the camera will appear to rotate counter-clockwise. Move the world right and the camera will appear to move left. So, when the user tells the "camera" to "move right" you don't want it to "move left". So, everything in the view matrix has to be backwards compared to the world matrices of the individual objects.

But this allows your camera to move through the 3D world.

And I can't help you with OGL at this point, but I've written a few DX11 coding examples to show exactly how 3D works with strongly commented source code.
Was This Post Helpful? 0
  • +
  • -

#7 kutuup  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 30
  • Joined: 01-August 14

Re: Struggling to understand how graphics work in terms of code.

Posted 25 February 2016 - 09:32 AM

Wow, that's one mother of an in depth explanation! Thank you so much for taking the time to write it up!

Your video series sounds like something I definitely need to watch, so I've set aside some time tonight to have an initial viewing.

From your breakdown of the process, it does sound like I had something resembling the right idea on how it all works, but it's incredibly helpful to have the gist of it all laid out for you like this. I'm gonna have to re-read this all a few times to take it all in, but I greatly appreciate you taking the time to write it!

I do have a question about how camera movement works, though. Does the camera really remain stationary in most scenes while the environment moves around it? It seems like an illogical way of doing things. Is it possible to get free camera movement around a stationary environment in most graphics libraries?

Thanks so much!
Was This Post Helpful? 0
  • +
  • -

#8 BBeck  Icon User is offline

  • Here to help.
  • member icon


Reputation: 792
  • View blog
  • Posts: 1,886
  • Joined: 24-April 12

Re: Struggling to understand how graphics work in terms of code.

Posted 25 February 2016 - 08:37 PM

Yep, it took me a better part of a decade to figure out how it all worked. I had to figure much of it out on my own with little help. Time spent in XNA with various books on that really is what got me there. From there I was able to use that knowledge to get DirectX 11 figured out. I've been hoping to jump to OGL, but just haven't had the time to dedicate to it this last year with things going on.

As for the camera, that's how the math works. Maybe my vector and especially my matrix video might make that a bit more clear. But this is the only way it can be done, because of the way the math works.

The scene always moves around the camera although hopefully the illusion is that the camera is moving through the scene.

Generally, when you're programming, you don't usually think of it so much this way. You use a world matrix for each model to place it in the scene and move it in the scene. That's completely separate from the camera. And you generally imagine the view matrix (camera) moving through the scene. You give it translation (movement) commands as if you are moving and rotating the camera. But if you get into the underlying math, the first thing you find out is that the view matrix is inversed because the math has to do the opposite motion of what you are telling the camera to do. And second, if you get into the math inside the view matrix you will quickly discover that there is no such thing as a "camera" for you to move. That view matrix is actually a mathmatical operation you perform on the models. There is no camera for you to instruct; the view matrix is applied to every vertex in the scene to move the entire scene around the origin where things will be "projected" to mathematically. Even the projection matrix, which projects the 3D vertices onto an imaginary 2D plane is actually a mathematical operation performed on every vertex in the scene rather than something you could truly call a "camera".

But you can think of the view matrix as a camera that you are moving through the scene. Even the inversion may be done for you most of the time. So, when "moving" the view matrix, it probably seems like you are moving a "camera" through the scene. But again, when you understand all the underlying math you realize there actually is no such thing as a "camera" for you to move. All there is is the vertices in 3D space and the fact that you are projecting them onto a 2D plane that's axis aligned and on the XY plane. So, the only thing that can move are the vertices of the objects because there is no such thing as a "camera" to move.

So, there are generally 3 matrices that control everything. There's the world matrix for each object (and when you get more advanced you'll have a world matrix for each part of the model). Then there's the view matrix which you can imagine being your camera. And then there's the projection matrix, which you can kind of think of as the camera lens but it's job is to convert those 3D points into points on a 2D plane that matches the shape of the screen so that it can all be drawn on a 2D computer monitor and look 3D.

But again, even though the scene is actually moving around the camera. You can imagine the camera moving through the scene when you work with a view matrix. I probably imagine the camera moving through the scene when I'm working with it. But reality is that the view matrix moves the vertices of the scene and moves the vertices around the origin which mathematically is the camera.

This post has been edited by BBeck: 25 February 2016 - 08:40 PM

Was This Post Helpful? 0
  • +
  • -

#9 BBeck  Icon User is offline

  • Here to help.
  • member icon


Reputation: 792
  • View blog
  • Posts: 1,886
  • Joined: 24-April 12

Re: Struggling to understand how graphics work in terms of code.

Posted 25 February 2016 - 08:51 PM

Another thing I might point out is that you start with a model, or many models, you make in a modeling program like Blender, Max, or Maya. You export that model in a file and the vertices of the model officially have the points specified in that file.

When you load that file up, the vertices of the model are where ever that file says they are. And generally, that's always somewhere near the origin. You don't want all your models to always be at the origin. You want to place them somewhere in the scene and you may want to move them around.

There's several reasons that you leave those vertices alone. You just don't mess with them. One reason is you might mess them up. Another is that matrix algebra is highly efficient and allows you to do some pretty neat tricks and the graphics card likes to do things this way.

So, you just leave those vertex positions as defined in the file. To place the object, you apply a world matrix to it. Using matrix algebra, the world matrix contains the position, orientation, and scale of the model which allows you to place it and orient it (my rule is to never use scale, but that's off topic - scale it right before you import it, is my rule).

By applying that matrix to every vertex in the model, every vertex will be correctly positioned and rotated into it's correct spot in the scene. One matrix stores the math to move every vertex in the model for proper placement.

The view matrix contains math to simulate the camera. This matrix is combined with the world matrix to move every vertex in every model in the entire scene to simulate a camera in the scene. But reality is that the vertices of the models are what's actually being moved by the view matrix.

The projection matrix likewise simulates the lens of the camera projecting the 3D image onto a 2D surface. But the reality is that the first thing you do in the shader is combine the model's world matrix, the view matrix, and projection matrix into a single matrix that holds all their combined calculations which is then applied to every vertex of the model before that model is drawn.

Also, be aware that OGL tend to do things slightly differently. Like what we call a pixel shader in HLSL is called a fragment shader in GLSL I believe. But I think most of the general concepts here should translate over. I think some of the details about the way matrices are used in OGL might be a little different. I haven't got deep enough into OGL to say how it all applies there.

Most of this stuff is math rather than computer programming. So, it's so low level it should apply to everything.

You'll hear talk of "model space" and "screen space" and "world space". All that stuff just confused me when I was first starting out and so I refuse to use those terms. I never learned them, although I'm pretty sure I now know what they mean. But it's just confusing terminology as far as I'm concerned and I'd rather say "The vertices of the model are in the same position as they were given in the file when you imported the model" rather than "The vertices are in model space." And rather than saying "The vertices must be modified in screen space." I would rather just say "Now that you've projected them onto a 2D plane, do your calculations at that point." So, don't get too hung up in the terminology. They'll say something like "You must translate the vertices of the model from model space to world space using a world matrix." and I'd rather just say "Your vertices are where they were as specified in the model data file and you probably want to move them into a specific place and orientation in your scene by applying a world matrix to all the vertices in your model." Same thing, but I think the second way of saying it is about 1,000 times more clear to someone just starting out. Model space is basically how it was done in the model file. World space is basically how it is after you apply a world matrix to put the model into the scene where you want it. And screen space is basically what you have once you project everything onto a 2D surface to be drawn on the computer screen. You may do calculations at any of those 3 points depending on what it is you are doing.

This post has been edited by BBeck: 25 February 2016 - 09:00 PM

Was This Post Helpful? 0
  • +
  • -

Page 1 of 1