I'm using Thrivecart with Sales Engine for the membership. Do I set up a new Thrivecart product for each course I want to sell as a one off that's in my membership?
Each course you want to sell as an individual course should be a separate product in ThriveCart. If want to offer them as a Membership, you can group them and sell them as another product.
Check out How to setup your e-learning platform selling your Courses and Content on a One-time fee as a part of a recurring Membership at the same time for more details on setting up.