Change inline assertions to debug_asserts

All of these were things that are (should be) guaranteed true no matter
what input is given to the API, barring unsafe operations on the data.