Rules are meant to be broken. This is especially true with fashion. Why not pair black with brown? Who says that guys can't wear red pants? Why not wear unisex accessories? Follow the wisdom of Nike; if it feels right, "Just do it!"
Don't be a conformist, be a reformist. It may take a bit of creativity, effort, and courage, but, at the end of the day, you will rest easy knowing that you stepped out on a limb to walk towards being more
YOU <---which is always a beautiful thing.