Yahoo Publisher Network = teh slow

The last couple of days I have noticed a huge spike in load times on my sites across the board. I just implemented a round robin banner system to alternate between Google and Yahoo! so I figured it was something with that. I refreshed my pages that were running the code a few times and I noticed the Yahoo! ads took a LONG time to finish loading. LONG meaning 1-5 seconds. I disabled the Yahoo! ads for now. I have been hearing around the internet that a lot of people have recently been accepted into the Yahoo Publisher Network so perhaps this influx of new peeps is what is bringing it down in speed.