Full Rewrite

[thesuperzapper]

This is a complete rewrite of the CSD.

Highlights:

• Rewrote README.md
• Support for Airflow 1.10.3 (and all new configs)
• Template based airflow.cfg generation
  • I truly believe this is more maintainable than the crudini approach and allows easier expansion to webserver_config.py.
• Secure configs stored in environment variables, rather than airflow.cfg
• Cloudera Manager can now properly start/stop the service
• Cleaned up all .sh files into control.sh and common.sh
• Removed RabbitMQ roles (as users should set up RabbitMQ separately if they want to use it)
• Validate SDL automatically with mvn package using the Cloudera schema-validator-maven-plugin
• Moved to airflow linux user rather than root
• Fixed licence headers

NOTE:

• I have only tested this with the PR 3 of the Airflow Parcel
• I have used src/_aux instead of src/aux to allow Windows users to build the jar
• I can re-add the Makefile if you want, but since I am using the maven plugin for SDL validation, there is little benefit

TODO:

• I have included the following code in control.sh

  unset PYTHONPATH
  unset PYTHONHOME
  

  • This is because the current iteration of PR3 for the parcel mistakenly sets them. (This causes issues with spark-submit from the worker roles, which tries to use airflow's python instead of the base Cloudera one)
  • This can be removed once I have update PR3

---

Please discuss. (I am more than willing to make changes if needed)

[thesuperzapper]

@rssanders3 @razorsedge @Naveen481 are you interested in these changes? Or should I make a fork?

I recommend looking a the docs/readme here: https://github.com/thesuperzapper/apache-airflow-cloudera-csd/tree/full_rewrite