Ops have the ability to change channel modes on the channels they are on. They also have the ability to kick users and set the topic. Channel modes control who are ops and who is banned, among other things.
When one joins an empty channel, chanserv gives that user op. That op can set any mode that ops can specifically set. Still, if somebody enters, and you leave and rejoin, nobody will have op. To fix this, if you want to run the channel, register it. Upon registering, you get founder. Founder has the ability to change any channel mode.
The following types of channel modes, applied to users, allow them to change other channel modes: Halfop (+h nick), Op (+o nick), Admin (or Super-Op) (+a nick), and Owner (+q nick).