Its pretty obvious virtually all flames are the energetic reaction between a fuel and an oxidizer. You understand rightly that both fuel and oxidizer are consumed, joined to create something else.
But since we live in an oxygen background, oxidizer is usually naturally present. Fire then becomes a likelihood only when fuel is added to your situation. So we started saying "flammable gas" for a strong or hi-energy fuel gas.
Oxygen is a strong or hi-energy oxidizer, true, but it is an oxidizer, so it is not a fuel, so they do not call it flammable.
Personally, I like your intuition better. It is like a 1000v battery. It is all part of a dangerous thing. No point in saying the negative terminal is not "High Voltage" just because it is not the positive terminal.
Actually, oxygen is the positive terminal, and fuel is the negative terminal, in a fuel cell. A fuel cell runs the same reaction as fire, just better controlled.
The point is, "reaction" is a purely bipolar relationship. Fire requires chemical opposites to happen.
As our use of technology grows, it seems that man's activities may at any time present us with any hazardous situation imaginable within the laws of science, regardless of its unlikelihood in natural surroundings.
We now have situations where fire starts because powdered oxidizers or pure liquid oxygen is released into piles of fuel. Inverting the earlier natural situation of a fire.
So oxygen really is as distinctively important as fuel, in making fires.
Oxygen is almost as prominent as flammable gasses, on our safety signs at work. In 100% concentration and under pressure, it is a "Hazardous Oxidizer".