The Information Security Talent Search (ISTS) is a three-day red/blue team security competition hosted at the Rochester Institute of Technology by RITSEC (The RIT Security Club), formerly RC3/SPARSA. The club held the 17th annual ISTS in RIT’s Student Innovation Hall on the weekend of February 23rd, 2019. Thirteen teams from New York, Maryland, and Virginia participated in the competition.
I joined four of my friends from various majors and year levels at RIT for our first ISTS blue team experience. We were Team #4, the systemd(emons) (“system demons”), a name coined by Brendan as a mash-up of the popular Linux init system systemd. This was a bit of good-natured prodding against Cucumber Linux, one of the many operating systems used in the ISTS competition infrastructure, but one of the few opting not to use systemd. As Scott Court (the primary developer of Cucumber Linux) was an ISTS white team member, we figured that he would enjoy our team name.
Drawing on our experience from last year’s Incident Response and Security Competition (IRSeC), we spent the two weeks leading up the competition developing our strategy, keeping several important components in mind:
- Divide the work — High-intensity competitions such as ISTS are designed to maximize the number of different tasks teams are expected to perform. Between keeping 11 servers, 2 clients, and a router secure and operational, performing injects (tasks and reports completed throughout the competition), holding King of the Hill boxes, competing in a Capture-the-Flag competition, and electioneering, we knew that the black team definitely cut our work out for us. It is important for each team member to have several roles and objectives to fulfill to avoid confusion and downtime.
- Use a central infrastructure—One of the most time consuming tasks during the start of the competition is changing all of the passwords for the scoring and administration accounts. Our team used LastPass on the local PCs to generate secure passwords, organize logins, and sync them among team members. We also noted the times passwords changed, who changed them, and why the change occurred to avoid spending time asking “What’s the password for <x> server?” or “Why did we change the password to <hunter2>?”. This also reduced the chances of the red team or other blue teams viewing or using our passwords.
The second most time consuming task is downloading and installing any extra services or features on the competition boxes. To solve this problem, Sergio set up a file server that we used to
scpall of the tools we planned to use for each box all at once.
- Uptime isn’t everything — While uptime does contribute heavily to a team’s overall ranking, it’s important to remember that injects carry essentially the same weight as uptime and that other activities not central to the competition can easily move your team one or two places ahead in the rankings. It is important that a team is well-rounded and has specific goals set to meet uptime and other requirements. Keeping a majority of services up consistently is excellent as long we can recognize when uptime is sufficient and our team members can spend their time on the other activities as well.
- Create a 5-minute plan — The competition infrastructure is most vulnerable within the first five minutes when blue teams have access to their boxes and red team can begin attacking them. It is very important to ensure that each team member knows which boxes they are going to secure, what steps they are going to follow, and when to go on the offensive. We made a 5-minute plan for each operating system, added steps specific to each service that the box is running, and assigned each plan to a team member that had experience in administering that subject.
The Night Before
The teams met in the Golisano auditorium Friday afternoon prior to the start of the competition for the blue team briefing by Sean Newman, RITSEC’s competition architect, the keynote speech from Lennart Koopmann, founder of Graylog, and introductions from sponsors and the red team.
After the briefing, my team began finalizing our plans for the start of the competition that following morning. We divided our roles by platform. Emily and Brendan would take the Windows Server boxes for familiarity throughout their first security competition. Josh took half of the UNIX-like boxes, and I took the other half for my first time securing Linux in a competitive environment. Sergio had experience with Palo Alto Networks devices, so he would take the router and also assist with Windows. We then divided up the other activities, also sorting by strengths and familiarity with the subject matter or technology.
The black team revised the blue team manual shortly after we made our plans. The topology didn’t change significantly, although it did cause concern at the start of the competition and slowed us down as we reassessed our roles. Always be sure the documentation you have is as current and complete as possible!
We also reviewed our five-minute plans. Samples for our Windows and Linux boxes are below.
=============Install LastPass to bench PC (NOT VIRTUAL ENVIRONMENT).Connect to SSH (first-last@charlie.<redacted>.com) in virtual to download:
- <A friend's program>
- <Another friend's program>
- <Another nice collection of programs>
Change passwords for administrative users (LastPass generate and save):
- Open Active Directory Users and Computers
- Change default passwords for any accounts that can be logged in to
- Check for suspicious accounts and disable (confirm accounts with White Team if necessary)
- Check for users that have more than the necessary permissions
- Take a registry backup (SSH)
- Make a backup of any important configuration files (SysWow, cmd, etc.)
Check for services, programs, and autoruns at startup
- Services.msc - suspicious services automatically starting
- Installed programs - suspicious programs
- Autoruns - Sysinternals tool (anything red or yellow)
Installed programs and running services, registry keys
- Process Explorer - options > verify signatures and check VirusTotal.com (don't leave this open or it will eat up all your RAM)
- Process Monitor - anything suspicious (registry and network)
- Add and Remove programs - suspicious, vulnerable, programs
- Turn Windows Features on or off - old, vulnerable, or unnecessary services (SMB 1.0, Telnet)
- Check for suspicious DLLs and use <nice collection of programs>
Windows Defender (if applicable)
- This will probably be red team's first target
- Turn on Windows Firewall
- Turn on RTP and CDP
- Controlled Folder Access
- Memory integrity (if no virtualization - may cause issues)
- Run Windows Update (if applicable)
- Update any vulnerable applications that are still used
Secure running services
- Inventory Windows Server manager for running services
- Enable logging
Re-change administrator password and save to LastPass.
=============Install LastPass to bench PC (NOT VIRTUAL ENVIRONMENT).Connect to SSH (first-last@charlie.<redacted>.com) in virtual to download <things>.passwd - assume compromised, but change anyway vim .bashrc - remove all backdoors found there .bash_profile /etc/bashrc - find location of global bashrc files/profiles lsmod - check permissions on important filesdmesg - tainted kernel shows signs of rootkit cd /etc/systemd/ - look through ALL service files on the machine - there are several places htop - look for shady processes for web - look for backdoor shells - /etc/apache2/sites-enabled netstat -antpl: look for reverse shells cat /etc/passwd > homedir/passwd_old, set attributes as read only look for shady users - disable ones that aren't necessary, or change password rootkit check - check sshd binaryre-change password
The campaign began at 9:00 AM on Saturday and resumed at 9:00 AM the next day. Caterers provided breakfast and lunch, and sponsors such as Palo Alto Networks and Cisco held “firetalks” during breaks and downtime. Teams were also able to discuss strategy and compete in the CTF during this time.
Start of the Competition
While implementing our five-minute plans, we took several precautions with regard to the red team and other offensive blue team members, many of which were successful. Others were not, and some even backfired on us.
- Hashing and Backups — Thinking back to IRSeC, the Windows team took backups of the registry on each system and both teams took backups of binaries in several system folders that another team was likely to modify or backdoor. Sure enough, an offensive blue team prevented most other teams from running the
netstat, or similar commands on Windows systems early in the competition. I was able to restore these commands, and Josh was able to do similar with the
aptcommands that were disabled for him soon after.
- File Names — Malware will occasionally not allow users to execute certain programs used for its detection, such as the programs included in the SysInternals suite. Renaming these applications typically circumvents this issue.
- 2-Factor Authentication — We suspected that red team would place keyloggers on most of the boxes as a method to maintain persistence or allow access to services we would be using throughout the competition. Indeed, custom Duo authentication set up by Sergio prevented an unauthorized party from uploading files to our remote file server on the first day of competition.
- Antivirus — We shot ourselves in the foot in this area. Most consumer-grade antivirus can easily be disabled by any user on the system, so we deployed Sophos Endpoint Protection on the Windows boxes. At the end of the competition, red team ran an Ansible playbook that repeatedly deployed the Memz Trojan to all blue team Windows PCs. While Sophos blocked and removed the infection, it then restarted the box it had cleaned. As the malware was continuously re-deployed, the box kept restarting. This sent our servers into a bootloop, but it was fortunately within the last 5 minutes of the competition, so the impact was minimal.
Injects were worth 32% of the overall score, second only from uptime (33%). White team distributed injects via team emails on the ProtonMail service, sent out periodically during the competition from various campaign “departments”. Teams could also complete bonuses to receive additional campaign funds.
Some injects we completed were:
- Set up a web app for campaign donations
- Inventory devices and services on the network
- Configure OSSEC
- Write access control lists (ACLs)
- Create an incident response report
- Analyze a malware sample
Other teams made several injects significantly easier when they accidentally used the reply all feature to submit their solutions! Moreover, Team 2 (DawgSec) was able to advance three places in the final scoring due to the amount of injects completed. Injects matter!
Scoring and Debrief
In the red team debrief, the red team members reintroduced themselves, talked about their careers and the companies they represent, and discussed hiring opportunities. They also went over common mistakes blue teams made during the competition, such as:
- Not changing default passwords for the scoring accounts — I believe this was mostly due to the intermittent availability of the scoring website throughout the competition, preventing teams from logging in and changing the scoring account passwords. Moreover, as most teams did change the passwords on the administrator/root accounts, I assume that some teams (likely individuals who have not competed in an attack/defend competition before) were not aware of how to use the scoring website properly. If this is the case, doing a quick demo of the scoring website features at the start of the competition could reduce the repetitive nature of this issue.
- Not checking for accounts that have administrative privileges or
sudoaccess — As the red team said, changing the root user password and disabling SSH as root does not make a system secure. In many cases, the scoring user
sudopermissions from the start. Combine that with a default password and the box becomes a gateway for just about anyone to run
rm -rf /, overwrite important binaries with
/dev/zero, or take other nefarious actions.
- Using backdoored programs or binaries — We were able to detect and fix some of these due to the hashes we took at the start of the competition and by monitoring for abnormal processes when running the likes of
passwd, but we undoubtedly missed several instances of backdooring and set off a few of red team’s traps. OSSEC will definitely be in my toolkit to implement immediately next time around.
- No ACLs/poor monitoring of network (HTTPS callback) traffic — Teams didn’t have access to the router on the first day due to technical difficulties, which might have something to do with this. It’s also very common to forget about network devices or not know how to configure them. Here’s a rundown on ACLs (for Cisco, but the concept is the same), and here’s a nice tool I used to link network traffic with process activity to find some of the malware that red team deployed.
- Rootkits —Despite possibly the
backdooraccount that I saw in the advanced recovery options (but not in Windows itself), I didn’t notice any rootkits. I also didn’t figure out how to remove that backdoor, though. Nevertheless, the red team said that rootkits were there, which means this is an area I am definitely going to work on while preparing for later competitions.
Now, there were a few random things I noticed that I’d like to talk about:
- Despite the memes, Cucumber Linux is really solid. I heard (not verified though) that Red Team basically became bored while trying to break into it and couldn’t really do much; to that end, props, Scott! 👏
- Lots of teams had paper password sheets. In my opinion, not only does this give other blue teams or red team (see: Jaime with a GoPro) the opportunity to shoulder-surf and profit, it’s even one of the most commonly covered bad practices in businesses today. This is essentially the equivalent of putting a sticky note on your monitor.
- Automation is key. Learning about the red team’s use of Ansible to obliterate Windows Defender and deploy malware was very interesting, and it seemed that many teams not only scripted their five-minute plans, but they also had offensive scripts for the first five minutes as well. I am going to keep this in mind and attempt to develop a more robust set of scripts for both purposes next year.
- The Plank of Shame — I know it’s a tradition and mostly a joke, but I still feel this goes against the RITSEC’s mission of building security “through community”. There’s no better way to ruin a community than to publicly ostracize people who are new to security with a last place award. I don’t like this.
- I’m never going to look at KFC the same way. Thanks, Chaim. 🍗
(Scoring breakdown: Uptime — 33%; Injects — 32%; King of the Hill — 20%; CTF — 15%)
1st. Team 2 — DawgSec (UMBC)
2nd. Team 10 — HTCPCP:// (UVa)
3rd. Team 8 — Can’t Crack This (RIT)
4th. Team 5 — USS Friend (UMBC)
5th. Team 4 — systemd(emons) (RIT)
6th. Team 9 — Eiffel 65 (RIT)
7th. Team 11 — The Tactical Swaglords (EMU)
8th. Team 7 — UB Geese (UB)
9th. Team 1 — UBnetdef Helpdesk (UB)
10th. Team 12 — The Gophers (EMU)
11th. Team 6 — Monsters University Dropouts (SUNY Albany)
12th. Team 3 — HYTTIOAOA (RIT)
13th. Team 13 — default_name (RIT)
I really enjoyed my time at ISTS17. While my team didn’t place, it was an incredibly valuable learning experience and one that I will definitely use to improve my skills for IRSeC this year and other competitions in the future. Thanks to the RITSEC black and white teams for the many all-nighters you put into this, sponsors for supporting us, red team members for taking the time to simultaneously love and hate us, and to RIT and the Computing Security department for providing the computers and Student Innovation Hall for our use.
All pictures of this event are available on RITSEC’s Facebook page.