What The Hack 2005
by Vasil KolevIt was incredible. Really indescribable.
(and I could just vilely stop here, with this excuse :) )
The event was mostly a gathering of ~3000 people at a camping ground in tents, where there were showers, normal toilets, food, drinks and 8-9 big tents (one of them could fit 1500 people inside) for lectures, bars, etc. There was a serious network (which I’ll describe belo, the schemes and some more detailed information can be found at wiki.whatthehack.org/index.php/FHQ). The core team wasn’t big, most of the work was done by volunteers, divided in teams, which was great, because not only everyone could participate, but a lot of work got done. I have no idea how this can be without the number of volunteers that were there (on this topic you might want to see Rop’s closing speech).
The event was closely pursued by an enormous quantities of rain, really enormous – it rained most of the time, the air was humid, dense, the tanks of the air conditioners in the NOC had to be drained every 8 hours, and there are even some photos of pumping the water under some of the tents. At 29/30 there was a big rain that made a lot of people to move to other places (especially the core team, who was at field K, which had no drainage). Some people slept in the NOC tent, I slept with the servers, and Walter had the experience to wake up and feel some electricity from the water on the floor in his tent. I wondered at some point if the people would change their bicycles for boats.
I wanted to describe all the people that I met, because they were really interesting, but it’s going to take too much space and time, and knowing my memory, there’s a great possibility to write something stupid, so I’ll just name them – Arien (core switches, full of ideas, Walter (head of the FHQ), Athilla (the main guy in the pulling the cables for the rings), Willem (tracking/monitoring switches in the field, Andre (field switches), Ernst (everything, this guy was unable to stay without something to do :) ), Reggie (also almost everything, always had a joke :) ), Maxim (head of the Wireless NOC, saved me one morning with a bottle of schnapps), Pim (who showed me how to configure a Juniper and didn’t scream at me for the things I did to the net :) ), Stephanie (words can’t describe her, when she smiled we had no need for the sun :) ), English Pete (he never stopped at one place, he was the adrenaline of the event :) ), Bassam Hassan (whom I couldn’t find at the event after his lecture to ask him a lot of stuff about Iraq, I’ll find his email from somewhere), and other known and unknown people…
I finally met Bine face to face – we’ve known each other from the time I started using Orkut, and had spoken about a lot of things, but seeing someone in real life is totally different. Too bad that both of us had too much work, and weren’t able to speak too much (she saved me from freezing by landing me her backup inflatable bed, for which I’m eternally grateful :) ).
What did a great impression on me was how many people are using Mac OSX – at least half of the latops were macs, and some people told me that I’ll switch to one someday (which I don’t feel believable, to be honest – I don’t like the OS, it’s not that nice to fiddle with :) ).
I’ll try to tell shortly about some fun moments in the event:
An optimist had ran arp spoof and wasa trying to make a ssh man-in-the-middle attack. He got caught in no time, and his photo showed up on the wiki :).
We filled the outside link at about 1/3 (looks like nobody had came to leech :) ).
A girl did a striptease in front of a webcam.
Someone had announced (jokingly) that the police officers that were at the event (they were 16) will be giving a lecture titled “Lawful interception”. A lot of people had gone there, the police guys were pretty surprised (here’s the entry in the Wiki, even AP fell for it :) )
In the end I was so tired and not really normal that I made a wireless cable and a wired/wireless converter (what they looked like I leave to your imagination :) ). That was mostly because a few days ago someone had managed to play a joke with an user, that he needs a wireless cable to use the wireless network… We gave as present one such cable to the wireless people.
In the last 2-3 days we used the NOC as a place to dry clothes – one of the ropes was the one that was used to pull the shutters, and the other one was an UTP cable, tied with cable ties.
When the police officers were going out on a round, the CERT car kept following them with it’s siren turned on. It was really amusing :)
With Athilla we were caught in the Megabit bar at some point and it turned out that there was only one free ethernet cable, so I connected my laptop and routed him through my wireless interface. It was a pretty fun exercise :)
Because of the weird laws there was no hard liquor there, but a lot of weed was smoked. Too bad I didn’t bring a bottle from Bulgaria, to make greyana rakiya… But Bobson managed to make a great dessert with vodka, dried cherries and chocolate, which we should test locally too. There were some schnapps-es, which were about 20% – which was good enough :)
One morning the water in the showers was pretty cold, and Pete joked in the Wiki that he did that on purpose, to better wake up the volunteers in the morning, to be ready for the work… I think that if he had really done that, we’d had him drowned in some ditch.
If you want more, the Wiki is a great source of information :).
The network setup was as follows:
The core network consisted of 3 Foundry BigIron switches (in the NOC, WhatTheBar and Megabit) connected with 10gbps links. The outside connection and all the inter-vlan routing were done on a Juniper M40, which also (at least for a while) was DHCP relay. In the field we used HP2626 and HP2824 switches, which with we had made two rings in a way that if power failed at some place it won’t be that big of a problem (the map is somewhere in the wiki :) ).
Logically we had some VLANs – one for the field (/19), one for the big tents (/19), one for the wireless users (/20), a few /24 for internal needs (administrative, switches, PXE for the wireless people), one for servers (/24), one for co-location (/23) and one for the OLSR people (/23 or /22).
For servers we had 5 machines for different services – pentium3 with small drives and a good quantity of memory. Two of them were HA cluster for some of the core services (DNS, DHCP, NTP), one was for Arien’s spongebob (a system for mitigating the arp traffic), one was for WINS server under Windows (which was done by Athilla), and the last one was for the switches’ administration (syslog, tftp, etc.). All of them were running debian except the windows machine, which I had configured, tuned, etc, they had ipv6 support and had almost no problems (there was some stuff around the dhcp, but it turned out to be because of the Juniper’s relay – too bad Pim wasn’t there, to do all possible tests). The only thing that troubled me was that all of the servers were front-mounted only, and I have no idea how did they stay there, they weren’t light.
(we could’ve used more power, then we could’ve hosted the wiki)
There was one machine from RIPE for snort, which had it’s own optical interface, and received mirror from the outgoing port of the juniper. The machine had a lot of problems, because FreeBSD 5.4-RELEASE couldn’t handle the load, and 6.0 died every 15 minutes, I have no idea what happened in the end.
(about this machine there was one really fun (for me) moment, one of the OpenBSD guys had came into the NOC tent and we spoke a bit on this problem, so he said that the polling of an interface was a really stupid idea (and that’s why it isn’t implemented in OpenBSD), and I asked him – ok, what if you have 100k interrupts per second? He said – yes, but what to do in the cases in which you don’t have so many, and I answered – you just switch between the two modes at different loads, like in Linux. We gave me an strange look and left, and I had this evil grin for half an hour.)
There was one machine brought by Pimto monitor the traffic on the interfaces – it was running only cacti.
The wireless people had 2 machines co-located – one for monitoring and one for dhcp and stuff, two big SUNs.
There were a few more SUN servers with a shared storage, but I never understood what it was used for.
We had some machines co-located by users – 3 in the NOC (one big HP that we used as a table, that had HP-UX running and was a shell server, one mac mini with OSX, which had an ircd connected to IRCNET, and AMD64 machine that was indexing the local FTPs, binary newsgroups, and had some vservers running (one of which was for the local OpenVPN server)), 2-3 in WhatTheBar (from archive.org, red 1U boxes).
All servers had separate IP addresses, and all services had addresses too, which made the moving between servers really easy, it also helped a lot with the heartbeat configuration.
We had full IPv6 support and routing.
The network had some fun moment, like PEG-DHCP. To the requesters we gave pegs with the last two octets of the ip address and a piece of paper with the rest of the configuration (IP prefix, DNS, etc.), which guaranteed that they have an unique address and weren’t dependable on the DHCP servers (which had it’s problems). The idea even has it’s own RFC… In the end because of the rain people were coming to get pegs to hang their wet clothes, and on the last day a guy came to get some of them to sell on ebay…
We had 15 km cable, from which we used about 3 km for the rings and for other stuff, the rest was used by the people for their connections in the field – we finished 11 km for one or two days. The principle was as follows – we had tools, jacks and cable, everyone could make a cable, and who wasn’t able to do so, had one crimped by us, for one beer (we accepted energy drinks, too). We drank a lot…
The organization had it’s flaws – some things weren’t clear before going there, we didn’t have clear policies for some things and twice I got chewed for doing some things. The design of the network was done there, and a lot of things weren’t clear, because some of the people that made the original design weren’t there, and we had a lot of misunderstandings, which we managed to clear :) We in fact didn’t have a problem that we couldn’t solve – in the beginning we even had a fiber cut – turned out that the Dutch railways went on to mow the grass with some enormous beast, and managed to remove about a meter of the cable (I can’t find the photos from the fixing, they’re worth seeing). Then, in the rain tools were found and some people went to splice the cable (we were worried that they could drown in the ditches, the rain was that heavy).
As a small conclusion – I found a lot of interesting things there, like that I’m able to sleep while it’s light, at cold, wet, in a room with 20 machines and three air conditioners, to sleep for 3 hours and again to be able to do my job, to walk in a heavy rain and not care how wet am I – in general, what doesn’t kill me, makes me stronger (and the whole event was almost murderous). Combining this with all of the information, problem, solutions, etc, I think I know why is it done only once in 4 years…
And I’ll go again in 4 years, if I’m able.
TODO:
To ask Maxim on the IDS he recommended.
To finish the mirror of all video recordings.
To do a copy of the wiki because of the rumor that it will go away in a month.
To remember some other interesting moments that I’ve missed here.