Setting up Antialiasing

GL_ARB_multisample defines the mechanism for antialising in OpenGL.

In order to use antialiasing of any kind, you must create a proper pixel format for your rendering context. This is a platform specific process. Furthermore, in order to check for multisampling support, you must have an active rendering context. Thus it is advised that you first create a dummy context that every GPU should support. Once this dummy context is initialized, you can check for antialiasing support and destroy the dummy context. If antialiasing support is available, you may then create a pixel format that uses antialiasing.

In Windows, the following attributes must be added to your pixel format attribute array in order to use antialiasing: WGL_SAMPLE_BUFFERS_ARB and WGL_SAMPLES_ARB. WGL_SAMPLE_BUFFERS_ARB should be set to 1. WGL_SAMPLES_ARB should be set to the number of samples you/the user request (2, 4, 6, 8, etc.)

In MacOS X, you can request antialiasing through Cocoa, AGL, or CGL depending on what your application needs.

The following attributes must be added to an NSOpenGLPixelFormatAttribute array : NSOpenGLPFASampleBuffers and NSOpenGLPFASamples. These directly correspond to WGL_SAMPLE_BUFFERS_ARB and WGL_SAMPLES_ARB.

If you are using AGL, the following attributes must be added to your attribute array: AGL_SAMPLE_BUFFERS_ARB and AGL_SAMPLES_ARB. These directly correspond to WGL_SAMPLE_BUFFERS_ARB and WGL_SAMPLES_ARB.

If you are using CGL, the following attributes must be added to your attribute array: kCGLPFASampleBuffers and kCGLPFASamples. These directly correspond to WGL_SAMPLE_BUFFERS_ARB and WGL_SAMPLES_ARB.

Once a context with a sample buffer and a number of samples greater than 1 has been created, antialising can be activated with the following command

glEnable(GL_MULTISAMPLE_ARB);

References:
Multisampling specification
Multisampling in Windows
FSAA in Mac OS X