When a dowel is slightly undersized for the hole you’ve drilled for it, a scrap piece of fabric is a great way to shore up the fit. Spread glue on a small piece of fabric and the dowel. Place the fabric over the dowel hole, and tap the dowel down into place. Trim off the excess fabric before the glue dries.
– Serge Duclos