Post
106
A word of warning that my initial run of 3.6 produced unfavorable results vs 3.5; performance on out-of-distribution and instruction following tasks appears to have collapsed. Potentially a vLLM 0.19 issue here, the original eval was done with their fork of 0.18. I am re-running both with nightly so we have apples to apples and will report back, curious to get to the bottom of this one.