Hi
Yes, although I think it's a pretty common problem, well at least on the later XB models.
I own a 10-plate Uly XT, what I did was raise the back wheel and remove the lower belt guard from which you have a small area to work within where the pulley is not in contact with the belt.
I then could turn the wheel and remove all the loose flaking paint using a wire brush and some reasonably course rubbing down paper until I was satisfied I had a good key for a fresh coat of paint.
Firstly I used some quick drying primer which I brush painted over the bare metal areas, some of the original paint didn't come off in the initial prep which left these areas off noticeabily lower, the second coat of primer was to cover all of the pulley, and finally I brush painted a coat of matt black, which actually turned out more like a satin finish, over the entire pulley it now looks like new.
By working in the small belt free area it ensures you dont get any paint on the belt, also it saves the need to remove the wheel.
P.S. Intially I started by trying to carefully avoid painting over the pulley retaining bolt heads but gave up in the end and painted the heads of them also which actually looks fine.