In that time this setup has built a helluva lot of software, both open-source libraries and closed-source moonlighting apps, and I've learnt a helluva lot too. Time to share.
John's Continuous Integration Rules
The pipeline/flow should kick off the instant something is pushed to master. Waiting 59 seconds because we just missed the poll is wasteful. If we're using a modern source-control system, there is absolutely no reason to be periodically polling it for changes. It's the 21st century, last century's batch processing techniques aren't useful here.
Clean, Tagged Builds
The build must begin with a clean to ensure repeatability. Each and every successful build should be appropriately tagged so that the correlation between git commit ID and Jenkins build number is evident.
Versioning Includes Build Configuration
Things go wrong. Jenkins configurations get accidentally broken. It should be just as easy to roll back a misconfigured job as it is to roll back a bad code change.
If It Passes The Tests, It's In test
Yes the test environment will be volatile, but as long as the tests are good, it should be good-volatile; aka latest-and-greatest. This puts the onus on developers to write comprehensive, meaningful tests. The test environment should be a glittering showcase of all the awesome that is about to hit prod.
Fully-automated push-button test/staging-to-prod
No manual funny-business allowed. Repeatable, reliable, and (ideally) rollback-able from the Jenkins UI.
Desired SetupThe simplest thing that will deliver to the target environments, and abides by the above rules:
[User] | commits to master | v [BitBucket] | pokes | v [Secured Jenkins Instance] | commits to | v [Heroku TEST/STAGING Env] (manual trigger) | v [Heroku PROD Env]