I doubt exposure to much of any native speaker who has even passable grammar and syntax would be doing more harm than good, especially for the beginning speaker. Exposure and feedback, you're helping them a lot by offering to help.
Speaking is usually one of the hardest and final parts of language learning to come together for those taught formally, because it requires creation of phrases words and immersion with consistent feedback. How many written tests, readings, and monologues in traditional classrooms are structured this way.
Since you're the native speaker, I would just start speaking. You should be able to pickup on their level of speaking and adjust yourself to it. When you talk be mindful of what you say to aid in listening. Try talking at just a pace faster than they are speaking. It can be easy to slip into "normal" conversation patterns you'd have with those you spend time with which are fluent. Try to keep from doing this best you can.
Try to have guided topics to discuss to help introduce new words and phrases related to that subject. Hell, role play some common day-to-day life event they may struggle.
Another suggestion, for "sticking points" in the conversation, maybe have a laptop or tablet that can translate key words and phrases they might want to say in their native language, but can't express since they forget or are unsure of the translation. Then you can help them work through what they want to say with questions and feedback.
Lastly, I'd suggest patience on your end with repetition of phases/ sentences. On the other side of the coin let the speaker who's trying to learn not feel any shame in stopping to ask for your to repeat or explain. If they're shy about doing this you may assume they understand more than they actually do. Put the speaker at ease with your ability to be patient and helpful.