Notes from RH Summit/Ansiblefest 2024

Reliable Network Compliance with Ansible DevSecOps

1. user changes template/script
2. change triggers pipeline
3. security analysis
4. build config
5. apply config
6. run validation tests
7. publish artifacts

Automated App Remediation using EDA

observe, evaluate, respond

• Monitoring: something happened
  

• Observability: why something happened

ansible rulebooks - use if-this-then-that rules - continuously running

SSH CA Authentication w/ Hashicorp Vault and RH AAP

• problem
  • leveraging keys at scale
  • key rotation is difficult
  • mishandling private keys
• solution
  • using an SSH CA
    • PKI-like infrastructure
    • designed for SSH
• SSH certs are signed with:
  • SSH CA priv key
  • TTL
  • roles
  • other SSH instructions
• Vault as SSH CA
• user authenticates to vault before auth’ing to server
• Vault integrates with AAP
  • secrets lookup
  • signed SSH

Security-as-Code and Firewall Policy Automation

• aims to add visibility
• move from spreadsheets to data structures
  • direct translation to policy
• implmenetation + backout
• using servicenow + github + AAP
• request form + approve + implement

process: 1. discuss the design 2. create automation diagram 3. write individual playbooks

building automation: 1. write steps as playbook comments 2. replace comments with code

• use “set_facts” to label the current stage for debugging
• use “set_stats”
• use failure handling
• modular design
• custom python modules
• firewall config caching
• compliance enforcement

Good Practices for Ansible

• used in lint validated content

examples: - make big items out of smaller items - use “foo_package” instead of “package” - use snake case for vars - use two underscores for internal variables - use a single source of truth for inventory - filter at source, not inventory (so use API to filter) - treat inventory in controller as disposable - define inventory as dir structure instead of a single file - make script to gen inventory dynamically - split long lines into multiple lines - yaml-multiline.info

call to action: - read the GPA - apply and share - contribute to improve GPA

Ansible Journey at Norfolk Southern

• major rail company
• previously using bash + puppet + chef
• management wanted to improve automation skills
  • made sure the team had the required time (1hr/wk_
• went through Ansible for DevOps
• learngitbranching.js.org
• learn to give/accept code review
• quality merges
• start developing standards
  • naming conventions
  • created internal docs
    • used for code reviews
• used mailing list support
• encouraged attending Ansible meetups
  • useful to prove skills are transferrable
• create detailed inventories w/ tagging
  • DC
  • dev/prod
  • hardware
  • DR priority
  • etc

gotchas:

• rhel 5 still in use (lots of hard-to-update edge devices)
• required keeping a copy of Ansible 9.12
• handlers + roles + playbooks behavior was a surprise
• idempotency is hard
• slowly changing customer requests
• disk mounts on existing servers

testing: - RHEL upgrades require playbooks to be reviewed + retested - getting other departments required managemnet buy-in - importance of culture of growth - replaced 800 line startup script with playbooks and roles

Ansible Keynote

southwest airlines

• ansible allows shorter maintenance windows
  • SWA does not perform maintenances while planes are in the sky
• side note: presentation was very scripted. “video of looking at SWA website for a flight Oh, didn’t realize you could see my screen”
• using ansible to create golden configs for network devices
• custom NOC solution using ansible?

jp morgan chase

• 10k unique playbooks
• secured, controlled, auditable
• post-conference note: pretty sure they said they have ~65M executions

Podman Desktop

• supports k8s and OS bases
• support for compose and pods
• “kind” app as opposed to k8s or docker?

RHEL Satellite Advanced Topics

• using ansible to
  • enable RH repos
  • add custom repos
  • create activation keys