Then you should have gone with a GitHub App instead of an Oauth App. The difference is that a GitHub App allows granularity of selection.