I'm making a scene builder in LWJGL. Objects in the scene have positions and rotations. Like most scene builders/modelers, I have colors handles to show the objects orientation. I have a typical setup, red points in the positive x direction, blue in the positive z.

The problem is the handles don't point in the correct direction. I have attached a screenshot showing the issue. The cube on the right has a rotation of 0, 0, 0, and the handles are correct. The cube on the left has a rotation of 0, 30, 0. Where I'm confused is why is the blue handle rotated 30 degrees clockwise, and the mesh is 30 degrees COUNTER-clockwise?

I compute the cube's rotation with

public Matrix4f getLocalMatrix() {
    // I update the position, rotation and scale directly, so I recalculate the matrix every time.
    return this.localMatrix.translationRotateScale(this.position, this.rotation, this.scale);
}

And to draw the handles I use

Matrix4f m = gameObject.getLocalMatrix();
gizmos.drawRay(gameObject.position, m.positiveZ(new Vector3f()));

...

public void drawRay(Vector3f start, Vector3f direction) {
    glBegin(GL_LINES);
    glVertex3f(start.x, start.y, start.z);
    glVertex3f(start.x + direction.x, start.y + direction.y, start.z + direction.z);
    glEnd();
}

I use a simple vertex shader, I don't think that's the issue,

layout (location=0) in vec3 inPosition;
layout (location=1) in vec2 texCoord;

out vec2 outTextCoord;

uniform mat4 projectionMatrix;
uniform mat4 viewMatrix;
uniform mat4 modelMatrix;

void main() {
    gl_Position = projectionMatrix * viewMatrix * modelMatrix * vec4(inPosition, 1.0);
    outTextCoord = texCoord;
}

The uniforms are being set correctly (I'm assuming). modelMatrix is set to gameObject.getLocalMatrix());

The only thing I can think of is some of my code is using right handed coordinates, and some is left handed?