I think you should be fine. Unless you find your self starting to eat more than normal or being drawn to them. In my case if I find myself looking forward to them it is not a good thing.

Personally I find I do better if my brown is something other than flour based. I like potatoes and brown rice and stuff like that. I started adding in breads that were whole wheat (because I found them convenient)But wow have I gained some weight so I am trying to rework my browns. Plus I feel much more solid when I stay away from the flours as my brown.

Kari